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V. 


i NF, ATR 


November 1, 2008 


Dear NEAIR Colleagues, 

Welcome to Providence and the 35 th Annual NEAIR Conference! 

The 2008 Conference Planning Team, led by Program Chair, Cherry Danielson; Local Arrangements Chair, Cate 
Rowan; Associate Program Chair, Leon Hill; Pre-conference Workshop Coordinator, Steve Thorpe; and Vendor 
Coordinator, Liz Deignan has put countless hours into this year’s program - organizing, coordinating and planning, 
right down to the smallest detail. This marvelously talented team has put together a wonderful program for you. I 
would like to take this opportunity to thank them on behalf of NEAIR for sharing their time and talents with us. I 
would also like to thank Beth Simpson, our dedicated and talented administrative assistant. Beth has been 
invaluable in this year’s conference planning process and has been immensely helpful to me and the team. 

The success of the conference is now in your hands as both conference attendees and presenters. We each need to 
engage actively in everything that the conference has to offer. I trust that you will enjoy this year’s theme “A 
Culture of Evidence: IR Support, Initiative, and Leadership” and I know that you will find the speakers, sessions, 
and networking to be positive experiences. 

What can you do over the next 4 days? O Attend sessions. O Present thought provoking material. O Ask 
penetrating questions. O Launch new ideas. O Challenge old assumptions. O Network with colleagues. O Take 
time to relax, refresh, renew, reenergize. O Continue the NEAIR tradition of extending a welcoming hand of 
friendship to conference newcomers or if you are new to NEAIR, introduce yourself to an “old-timer.” O Spend 
time with vendors. O Enjoy the entertainment. O Explore Providence. But most important of all, have a good 
time! 

NEAIR is a wonderful organization that is made up of talented professionals such as you. It has been my pleasure 
to serve as your president this year. If there is anything that I can do to make your conference experience 
enjoyable, please stop me and ask. I extend to you my best wishes for a great conference and an enjoyable visit to 
Providence. 

Sincerely, 


Denise A. Krallman 
NEAIR President 
2007-2008 
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AN ODE TO NORTHEAST AIR ON THE OCCASION OF ITS 35™ ANNIVERSARY 
CONFERENCE 


J. Fredericks Volkwein, Ph.D. 
Professor of Education 
Senior Scientist 
Penn State University 


In Williamstown, Mass, the year Nineteen Seventy-Four, 

Thirty-three colleagues gathered to open the door. 

Our own Lois Torrance was the Keynote Speaker, 

She was a model information seeker. 

While the annual presence of those Charter Members has dwindled from then until now, 
We still have one in attendance - Richard Heck, please take a bow. 

This organization has grown over the years, through thick and through thin, 

Blessed by good leadership from above, and by good colleagueship from within. 

For example, the list of national AIR Presidents 
Includes nine former NEAIR residents. 

And our administrative record appears rather pretty, 

Especially if you like governance by Steering Committee. 

Like the field itself, our conferences have been growing. 

It seems there is always something new worth knowing. 

And our conference Keynote speakers have been very hardy, 

Including the likes of Peterson, Ewell, Gallup, Zemsky, and Lombardi. 

I’m sure that this is no panacea, 

But we’ve had multiple conferences in each city that begins with the letter “P” 

— Providence, Princeton, Pittsburgh, Portsmouth, and Philadelphia. 

Our organizational profile appears quite robust, 

With official trappings building membership trust. 

We give research and travel grants as merit affords, 

Also Distinguished Service and Best Paper Awards. 

AIR connectedness provides helpful tech support 
And a wealth of publications to import. 

To show a proper environmental scan, 

We even have our very own strategic plan. 

And everyday, with each organizational breath, 

We should thank our lucky stars for a Simpson named Beth. 
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Has your Provost thrown you a curve? 

Seek help on the list-serve. 

Did your assessment plan flop? 

Check out an upcoming workshop. 

Do you need to respond to a budgetary winter? 

If so, come to us for a good mentor. 

Does your job security make you nervous? 

We’ll connect you to a career planning service. 

From projecting enrollments to outcomes assessment. 
Sharing our work and problems together brings refreshment. 
The challenges of institutional analysis, 

Also protect us from mental paralysis. 

So if accreditation is your recent obsession, 

Or if you need help with logistic regression, 

NEAIR is the professional resource of choice, 

Let it help you develop your quantitative voice. 

So let us unleash celebrations galore. 

Happy 35 th , may there be many more. 
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ASSESSING INSTITUTIONAL EFFECTIVENESS: 

THE MISSION ENGAGEMENT INDEX AS A MEASURE OF PROGRESS 

ON MISSION GOALS 


Ellen M. Boylan, Ph.D. 

Director of Institutional Research and Assessment 
Marywood University 

Office of Planning and Institutional Research 

Abstract 

As part of continuing research supported by the Teagle Foundation on mission and 
student learning outcomes, this paper describes intermediate steps in the development of a new 
performance measure for institutions called the Mission Engagement Index (MEI). The purpose 
of the MEI is to enable institutions to assess the relationship between their lived mission and 
student learning outcomes and, further, to allow a comparison of expected versus actual 
outcomes on institutional results. The data obtained to develop the measure is from institutions 
participating in an annual mission consortium administered in conjunction with the National 
Survey of Student Engagement (NSSE). The member schools appended a twenty question 
Mission Perception Inventory (MPI) to the NSSE and administered it one or more times during 
the period 2004-2007. Discussed are the size and strength of the data set, the statistical approach 
selected for developing the model, a process for determining the dependent and independent 
variables to be analyzed based on respondent and institutional characteristics, and possible 
outcomes resulting from statistical analyses. The Mission Engagement Index (MEI) subsequently 
developed will help to assess activity intended to advance institutional mission on college 
campuses, and serve as a reflection of the impact of the institutional environment on student 
learning outcomes. 


Introduction 

The mission statement of a college or university can be rolled up in a dry scroll and lost 
atop a library shelf or become words of vibrant inspiration that enervate the life and purpose of 
an institution. The latter circumstance is much more likely when clear alignment exists between 
an institution’s mission and goals and the ways they are realized in action. With regard to student 
learning outcomes, action takes form first in clearly articulated objectives for student learning 
and development (Chickering, 1993) and second, in the creation of an environment of instruction 
and activity that helps students achieve. Think of these actions as institutional inputs, however, 
and the question arises about how to measure their effect. The Mission Engagement Index (MEI) 
being developed here is intended to serve as a measure of the relationship between institutional 
mission and student learning outcomes. The theoretical framework and procedure for obtaining 
the necessary data to build the Index is described first, followed by a review of conditions that 
must be met before developing a statistical model, and then a discussion of possible uses of the 
MEI for tracking and comparing performance. 
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This research addresses the need of postsecondary institutions for adequate measures of 
mission effectiveness, particularly in relation to student learning. The fundamental assumption of 
the research is that a measure to assess student perception of the learning environment relative to 
institutional mission constructs can be devised and, further, that scores on these measures are 
valid indicators of institutional performance on mission objectives. A set of 20 questions called 
the Mission Perception Inventory (MPI) were developed in 2004 to query college students about 
perceptions of the mission of their college. The larger survey vehicle for appending and 
distributing the MPI was the National Survey of Student Engagement (NSSE), which is 
administered nationally each year to participating institutions. According to Pike, Kuh & Gonyea 
(2003), the NSSE was designed to act as a process indicator that could help colleges and 
universities see connections between programs and activities and student learning outcomes. 
Similarly, the MPI is a process indicator because its purpose is to assess activity on campus 
meant to enhance the learning environment and, at the same time, the context where learning 
occurs. Validity and reliability testing conducted in earlier studies (Boylan, 2005, 2008) has 
investigated and affirmed the ability of the MPI to assess and reflect the learning environment. 

An important assumption of this research is that sufficient cases exist for conducting the 
multiple regression analysis. Table 1 appearing later will indicate whether the required number 
of cases per independent variable has been achieved. Another important consideration is to make 
sure large standard error values are not observable in results by institution, and thereby assure 
that the characteristic of homoscedasticity (Glass, 1996, p. 180) is present. 

The purpose of this research is to develop a performance measure, or index, that will 
have stronger interpretive utility than MPI mean score results, alone. Using exploratory multiple 
regression, a prediction equation will be developed for generating residual scores on MPI scale 
results and individual question items. Residual score results produced by the Mission 
Engagement Index (MEI) will allow institutions to compare predicted versus actual outcomes on 
mission items, thereby providing a more meaningful way of assessing their performance. A 
critical step in the process of developing the prediction equation will be selecting for inclusion 
the most appropriate independent variables, or institutional characteristics, for the model 
equation. In view of this challenge, and mindful of the need for a sufficiently large and reliable 
data set for this analytical endeavor, this study intends to answer the following research question: 
Can a Mission Engagement Index (MEI) be developed to describe causal relationships among 
variables that affect mission perception? 

Review of the Literature 

Sources of information on assessment in higher education abound on shelves and 
electronic platforms as close as a click away. More difficult to obtain are assessment tools for 
measuring mission effectiveness in colleges and universities, particularly as defined by regional 
accreditation bodies. In addition to accreditation challenges, there are escalating stakeholder 
demands for accountability (Ewell, 2007), so providing evidence of fulfilling institutional 
mission can be a key part of assessment reporting. A statement of mission that is parsed into 
institutional goals becomes a much more manageable template for measuring performance and 
progress. 
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Frequently found in institutional mission statements are goals about providing 
professional preparation, striving for academic excellence, and developing a sense of 
commitment to community service. Toutkoushian and Smart (2001) suggest that a student’s 
experiences within the institutional environment correlate strongly with learning outcomes. The 
relation between institutional mission and student engagement and learning has been 
successfully measured using a methodology developed by Pike, Kuh, and Gonyea (2002). 

It is important for an institution to have clearly articulated objectives for student learning 
and development (Chickering 1993). If the objectives are agreed upon and emphasized in oral 
and written communication, and are evident in the articulation of programs, a stronger sense of 
the institutional mission can be present. Knowledge of shared mission can unify the educational 
experience of students and define purpose for them within the institutional setting. 

“Mission” is operationally defined for this research as the overall purpose and activity of 
an institution as articulated by the goals of the mission statement. This definition more accurately 
reflects the character and scope of activities engaged in by the institutions participating in 
mission research consortia, and is unlike the Carnegie Classification definition of mission found 
in popular publications like America ’s Best Colleges, the annual guide to colleges published by 
U.S. News and World Report, (2004, p.80). 

Research by Pike, Kuh, and Gonyea (2002) on institutional mission concludes that 
mission constructs can be measured. The methodology used by the National Survey of Student 
Engagement (NSSE) for producing benchmarks, or scales, of “effective educational practice” 
(Kuh, 2001, p.13) is reflected in this mission research in the way factor scales have been 
produced. 

An earlier limitation of this research having to do with reliance on a single closed consortium 
of participants for administration of the questions has been ameliorated. In 2008, a second and 
characteristically different consortium was added, allowing for further comparisons of outcomes 
by group. These data will be added to the inventory of study data for 2004-2007 consisting of 
more than 30,000 respondent cases obtained in 143 administrations from 83 unique institutions 
across the United States. 

Table 1: Count of participating institutions and 
respondents by administration year. 



Institutions 

First Year 

Senior 

2004 

15 

2,000 

1,827 

2005 

16 

1,279 

1,332 

2006 

24 

2,684 

2,854 

2007 

36 

4,533 

4,331 


91 

(83 unique) 

10,496 

10,344 
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Methodology 


The Mission Perception Inventory was developed using a mixed method qualitative and 
quantitative research analysis. The question items were derived from mission statements of 
colleges initially participating in a 2004 NSSE mission consortium. Scale-format questions like 
those used in the NSSE instrument were used for the mission research questions, as well. After 
verifying the clarity of questions with constituents, the instrument was administered by 
appending it to the main NSSE questionnaire. 

Giving evidence of mission effectiveness is requested of all member colleges by regional 
accreditation bodies (Middle States Commission, 2006). A deliberate effort was made to 
originate mission questions with equal utility for institutions of any affiliation or status. 


Table 2: Items by subscales and overall Mission Perception 
Inventory (MPI) by administration year. 


2004 2005 2006 2007 

Number of items in factors 


Sense of Mission 8 

Respect for Diversity 5 

Individual Actions 2 

Religious Practice 2 

Mission Perception Inventory 17 


10 10 10 

5 5 5 

0 0 0 

3 3 2 

18 18 19 


The factor analytic method used in previous analysis leading to the current research on 
mission has been described (Boylan, 2008). For each year of administration of the mission 
questions, factor analysis of results employing Thompson’s (2004) methodology indicates the 
presence of a highly reliable overall Mission Perception Inventory (MPI) scale consisting of up 
tol9 items and two to four subscales within the MPI (figure 2). Strong subscales contain multiple 
items, but caution is advised when interpreting factors consisting of just two items (Tabachnick 
& Fidell, 1996). 

The reliability of the MPI scale and subscales of Sense of Mission and Respect for Diversity 
were consistently high as measured via internal consistency coefficient alpha (Cronbach, 1951), 
as indicated in figure 3. To interpret the results of factor analysis, the rule of thumb is followed 
wherein only variables with loadings of .32 and above are interpreted (Tabachnick & Fidell, 
1996). The greater the loading, the more the variable is a pure measure of the factor. Comrey and 
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Lee (1992) suggest that loadings in excess of .71 are considered excellent, .63 very good, .55 
good, .45 fair, and .32 poor. 


Table 3: Cronbach's reliability analysis of the Mission Perception Inventory 
(MPI) and subscales 2004-2007 


Subscales 

2004 

2005 

2006 

2007 

Sense of Mission 

.87 

.88 

.88 

.90 

Respect for Diversity 

.85 

.84 

.87 

.86 

Individual Actions 

.67 

n/a 

n/a 

n/a 

Religious Practice 

.62 

.55 

.54 

.62 

MPI scale 

.88 

.89 

.90 

.91 


Until 2005, the National Survey of Student Engagement (NSSE) results by institution 
featured an Engagement Index showing predicted versus actual performance by institutions on 
the five NSSE benchmarks. The Engagement Index provided an added way of assessing 
institutional performance based on institutional characteristics and what might be expected 
considering a school’s profile. The change in reporting was based on the decision by NSSE 
researchers to reflect participant results on the student level, rather than by the average 
institutional scores that were previously the basis of comparison (NSSE, 2008). Observations 
over several years of administering the NSSE revealed that between-institution variance had 
been small compared to within institution differences among students, a problem for schools 
with larger standard errors. Larger standard errors are usually the result of a lower number of 
respondents (2008). Consider, however, that the NSSE institute conducts analysis on tens of 
thousands of cases of data in one year, alone, far exceeding the mission data collected to date. 
Therefore, for this research, where participating schools have smaller enrollments and 
institutional characteristics are similar, it can still be considered a worthwhile exploration to 
develop an Engagement Index for mission effectiveness to assist institutional assessment 
initiatives. 

An Engagement Index (El) was developed for first-year college chemistry instruction online 
(Trasker et al, 2003) with the purpose of measuring student engagement with different 
instruction modes. The independent variables selected for the regression model include student 
characteristics, design of online delivery of instruction, and performance behaviors. Age, gender, 
enrollment status, learning styles, cognitive preference, teaching style for presenting material, 
time on reading feedback, and performance assessment were all independent variables selected 
to predict “engagement” in learning. 
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Developing the Prediction Equation 


Independent variables to be explored for developing the Mission Engagement Index will 
be based on institutional characteristics of consortia participants. Two characteristics of 
participating institutions in consortia from 2004 to 2007 are illustrated in figure 1. 



figure 1: Consortia institutions by region and ubanicity 2004-2007 


The dependent variable for the prediction equation, or Mission Engagement Index (MEI), 
is the institution score on the Mission Perception Inventory factor. To determine independent 
variables and have confidence in the ability to conduct regression analysis, there must be a 
sufficient number of cases, or institutions, per each independent variable. According to Stevens 
(1992), “a recommended ratio of subjects to IV’s of at least 15 to 1 will provide a reliable 
regression equation.” With 83 unique institutions having participated, there is a sufficient 
number of “cases” for five IV’s. 

Table 4 shows the choices among independent variables that are available. Stepwise 
regression analysis will reveal the statistics on the variables which make the highest contribution 
to variance in the prediction and therefore make possible the determination of the most salient 
components of the index. 


Page | 9 


Table 4: Selecting independent variables (IVs) for regression analysis. 


Institution 

region 

Carnegie 

Class 

% 

resident 

% 

female 

0/ 

/o 

ethnic/ 

cau 

UG 

enroll 

% 

Part- 

time 

Setting 

% 

Accept 

a 

North 

U Master's 

39 

61 

95 

2119 

25 

Suburb 

74 

b 

M idwest 

U Master's 

33 

67 

70 

928 

40 

Suburb 

57 

c 

South 

National U 

32 

68 

39 

4130 

42 

Suburb 

54 

d 

South 

Bacc.. Coll 

43 

57 

76 

1011 

21 

Rural 

66 

e 

M idwest 

U Master's 

43 

57 

70 

1692 

23 

Suburb 

85 

f 

M idwest 

Bacc. Coll 

42 

58 

89 

978 

83 

Urban 

70 

g 

Midwest 

U Master's 

31 

69 

78 

2985 

37 

Suburb 

93 

h 

West 

U Master's 

32 

68 

23 

1096 

60 

Urban 

94 

i 

North 

Bacc. Coll 

92 

58 

62 

8568 

16 

Suburb 

42 

j 

M idwest 

U Master's 

28 

72 

91 

1502 

37 

Urban 

79 

1 

North 

U Master's 

24 

76 

79 

1320 

815 

Suburb 

60 


Discussion 

The conditions required for conducting regression analysis to develop a Mission 
Engagement Index (MEI) have required several years of consortium data collection in 
conjunction with the administration of the annual National Survey of Student Engagement 
(NSSE). This research promises to have utility and elegance for future applications as a result of 
several conditions that have been achieved: the internal consistency of the factors has been 
affirmed through repeat administrations and factor analysis of the data by year, the institution 
sizes relative to enrollment are reasonably varied and geographically diverse, and there are data 
sufficient to conduct regression analysis. It is concluded that a Mission Engagement Index can 
be developed and serve as a viable indicator of mission effectiveness for those institutions 
participating in mission research. 

Future steps include, first, an a priori selection of independent variables for the 
regression analysis and subsequent exploration of alternatives. Once the best selection of 
variables has been made, further exploration of differences among institutions can be conducted 
and conclusions drawn about characteristics of strongest influence on student learning outcomes. 


Acknowledgments: A two-year grant from the Teagle Foundation begun in 2007 makes possible 
the administrative and financial support to continue this research. 
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Introduction 

This paper takes an institutional research perspective on the interconnected 
phenomena of recruitment, retention, and utilization of faculty at research universities, 
with particular emphasis on the changing mix of tenure track and contingent (i.e., fixed 
term) faculty members. 1 In this paper, we have three objectives. The first is to present 
both broad national data and detailed information from a particular institution - Penn 
State University - on powerful forces that are prompting research universities to rethink 
fundamental strategies about the core academic workforce. Our second objective is to 
offer ideas about how institutional researchers can support the work of provosts, deans, 
and department heads to address challenges in faculty recruitment, retention, and 
development, and the optimization of faculty workload and productivity. Our third 
objective, frankly, is to bring the attention of institutional researchers to what is emerging 
as a substantial and possibly transformative change facing public research universities. 

This paper draws upon national data for United States research universities. 

While there is overlap among community colleges, liberal arts colleges, comprehensive 


1 A note on nomenclature: We use the term “tenure track’’ to refer to faculty members who have either 
been awarded tenure or who are eligible for tenure and who are working toward a tenure review. We use 
the terms “non-tenure track,” “contingent,” and “fixed term” interchangeably. 
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universities, and research universities, contextual factors are often specific to different 
academic market segments. We examine data and practices at Penn State in some depth, 
partly because we have access to the relevant data, and partly because it is useful and 
necessary to focus the discussion, and to provide examples of how institutional 
researchers can contribute at the institution level. 

The topic has a strong practical component, addressing questions that are relevant 
to both scholars and academic administrators in higher education. What can academic 
administrators and researchers do to better understand faculty labor markets and the 
structure of faculty work? How can they help strengthen, continually renew, and 
effectively utilize their institutions’ professoriate? 

Faculty Are the University 

There is a story that about 60 years ago, the president of Columbia University 
finished an address to the university community by presenting some of his ideas on the 
role of faculty within the university and better managing the faculty. A faculty member 
stood up and said, “Mr. President, with all due respect, the faculty are the university.” 
The faculty member was professor Tsung-Dao Lee, a physicist who won the Nobel Prize 
in 1957. The president of Columbia at that time was the former Supreme Commander of 
the Allied forces in Europe, Dwight D. Eisenhower. 

Another university president made an almost identical comment in an address at 
a different institution in the 1950s. That university president, who gave the faculty full 
recognition as "the backbone of every great institution," was Milton S. Eisenhower, 
Dwight’s brother and the president of Penn State from 1950 - 1956 (Penn State, 1996). 

There are reasons that these stories resonate, and why they provide a helpful 
starting point for this discussion of building, sustaining, and developing research 
university faculty. All organizations define and structure relationships with their labor 
force in some way - there is work to be done, and employees do it. However, faculty 
members in research universities are and are not “employees” in the same sense as in 
most other organizations. Academicians have considerable autonomy in deciding how 
they will spend their time and which courses they will teach, selecting research projects, 
choosing and evaluating their colleagues, and even influencing the criteria for 
performance evaluation. Faculty members are not necessarily “managed” in the way that 
we think of employees being managed in corporations or civil service or (as General 
Eisenhower learned) the military. 

In this paper we will draw on some information that the planning and institutional 
research office at our university develops and shares, through various mechanisms, with 
audiences such as the provost and vice provosts, deans, the faculty senate, department 
heads, trustees, and more. 

We will begin by very briefly highlighting findings from one of these analyses: a 
faculty exit study. 
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Faculty Exit Study 

There is a substantial literature on faculty satisfaction which spans decades (e.g., 
Caplow & McGee, 1958; Toombs & Marlier, 1981; Garshore, Hibbard & Stocking, 1983; 
Moore & Gardner, 1992; Collaborative on Academic Careers in Higher Education 2007). 
We cannot adequately review that literature here. But in brief, it strongly suggests that 
institutions can learn much about and act upon the determinants of faculty satisfaction 
and behavior. Effective practices should draw, in part, on an understanding of what 
matters to faculty members, and how institutional policies affect decisions about career 
choices, personal and family concerns, professional mobility, and productivity. 

For such reasons, in 1997-98 Penn State instituted an exit survey and interview 
process for all departing tenure track faculty members. That effort has been and continues 
to be led by the vice provost for academic affairs and supported by the institutional 
research and planning office. The goal is to better understand the experiences of tenure 
track faculty members and to respond to faculty concerns. 

The Penn State faculty exit study uses two data gathering mechanisms. One is that 
exit interview officers (current or recently retired faculty members) appointed by the 
respective colleges conduct interviews; additionally, confidential exit surveys are made 
available. Departing faculty members can choose one, both, or neither option. 

In any given year, about 120 tenured and pre-tenure faculty members exit Penn 
State, and about two-thirds typically participate in the exit study via the interview and/or 
the survey. Over the past ten years, 420 faculty members have participated in interviews 
and 368 have completed and returned surveys. The survey respondents include 167 
retirees and 182 individuals who indicate that they are leaving for some other reason. 

The findings of this process have in many ways been consistent with those of 
other researchers (such as Harvard’s COACHE project) already cited. In general, Penn 
State faculty members have been satisfied with their careers and with the policies and 
environment at this university. They have reported that institutional practices (on salary, 
benefits, workload assignment, support, and the like) have influenced - mostly in positive 
ways, but sometimes negatively - their satisfaction, ability to be productive, and 
decisions about leaving. Also, there are some essential differences across demographic 
groups (male/female, minority/non-minority, junior/senior), but the commonalities are 
greater than the differences. 

The exit study process asks faculty about many aspects of their academic lives 
(such as, for example, the climate for diversity, university awards and recognition, the 
quality of facilities, spousal employment opportunities, and so on). In general and in 
summary, the exit survey process finds faculty placing greatest importance on the 
following four factors: 

• An academically strong department 
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• Professional autonomy 

• Salary level and salary increases 

• Course teaching assignments 

We use Penn State’s faculty exit results as our first data point because these factors - 
that is, what our exiting faculty have identified as the most important dimensions of 
academic life - map well onto the changes and challenges explored in the remainder of 
this paper. The paper discusses what some observers believe is a transformation in the 
nature of the academic research economy, which may redefine what academic excellence 
means for public research universities. The paper examines the competitive market for 
top scholars, highlighting in particular the growing disparity between public and private 
university salary levels. And the paper provides data on the evolving nature of faculty 
composition and how that relates to substantial shifts in faculty instructional workload. 

The Evolving Nature of Faculty Composition and Faculty Workload 

There are approximately 1.3 million faculty members employed in United States 
colleges and universities. In 2005-06, according to data from the United States 
Department of Education (August 2007), 917,000 or 71% of these faculty members were 
in four-year colleges and 373,000 were in two-year colleges. Those proportions have not 
changed in a generation; in 1985, 71% of the United States faculty members were also 
located in four- year colleges. 

What has changed dramatically is the distribution of full-time versus part-time 
appointments. The data show 676,000 (of 1.3 million) or 52% of these faculty members 
holding full-time appointments in 2005-06 compared to 64% in 1985. The decline in the 
full-time share is even more dramatic when a longer time period is examined; 78% of the 
faculty in U.S. colleges and university were full-time in 1970. 

Tenure Track versus Fixed Term Faculty Appointments 

As recently as the late 1960s, fewer than five percent of full-time faculty hires 
nationally were for non-tenure track positions; today, over half of full-time hires are for 
non-tenure track positions (Finkelstein and Schuster, 2001 and Schuster and Finkelstein, 
2006). This represents a sea change in the structure of the professoriate in the United 
States. 


Page | 16 



80% 


60% 

40% 

20% 

0% 

1993 1995 1997 1999 2001 2003 2005 



— PSU Univ Park — ■ — Big Ten/AAU Main Campuses 


Source: IPEDS, December 2007. 

Figure 1. New Non-Tenure Hires as a Percentage of All New Full-Time Hires, 
Penn State-University Park and Big Ten/AAU Peers 2 


When we look at Penn State and its peer institutions, we also see clear evidence of an 
increasing reliance on contingent faculty. Figure 1 shows the extent to which this 
practice has taken hold at Penn State and at its peers. Until recent decades, hiring full- 
time faculty off the tenure track was exceptional rather than a common practice. 
Consistent with that national shift, the majority of new hires each year since at least 1993 
in these universities have been in non- tenure track appointments. 


2 

“ Member universities of the AAU - the Association of American Universities - typically benchmark 
among their respective main campuses, excluding medical schools, to optimize comparability; University 
Park is that main campus for Penn State University. 


Page | 17 


100% n 


80% 

60% 

40% 

20 % 

0% 



35% 


43% 


27% 


37% 


65% 



73% 


63% 



57% 
















. ■ Full-Time, Non-Tenure Track 
a Tenure Track 


1993 2006 1993 2006 

Penn State-Univ Pk BigTen/AAUDE 

Main Campus 


Source: IPEDS, December 2007. 

Figure 2. Contingent and Tenure Track Faculty, 1993 - 2006 


Since the majority of full-time hires have, since the early 1990s, been off the 
tenure track, it is not surprising that contingent faculty represent a growing share of full- 
time faculty. As shown in Figure 2, the proportion of the full-time faculty who are in 
contingent appointments grew in this time period from 35% to 43% for Penn State and 
from 27% to 37% for the research universities in its peer group. 


Faculty Workload 

The tenure track status of a faculty appointment naturally has an important 
bearing on the nature of the work being performed. Those appointed to tenure track 
positions are expected to develop active and successful research programs that 
complement their teaching, while those with non-tenure track appointments typically 
have more extensive teaching responsibilities. Moreover, teaching responsibilities may 
be further reduced for those tenure track faculty members who succeed at receiving 
external grant support for their research. The time needed to work on grants and 
contracts can come at the expense of time for teaching. External grants are competitive, 
and there are more opportunities for external funding in some fields than others. Thus, 
there is variation across disciplines in the average teaching workload for tenure track 
faculty members. 

Given the growing share of faculty members on fixed-term contracts, it is not 
surprising to see drops in the percentage of student credit hours being generated by tenure 
track faculty members. As represented in Figure 3, at Penn State, from 1999 through 
2006, the percentage of student credit hours delivered by full-time tenure track faculty 


3 We suspect that the higher percentages of contingent faculty at Penn State are due to differences in how 
universities categorize their faculty. At Penn State, many faculty members who are hired strictly for work 
on grants are non-tenure track faculty members. 
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declined from 48% to 40% (and annual research activity - mainly generated by tenure 
track faculty - rose from $393 million to $657 million). While many people may believe 
that graduate assistants and part-time faculty members are delivering a large and 
increasing share of undergraduate instruction, this is not true at Penn State. In fact, the 
proportion of teaching by part-time faculty and by graduate assistants (who are usually 
advanced doctoral students) has actually fallen. On the other hand, the proportion of 
student credit hours delivered by contingent faculty members rose from 21% to 32%. 



-Full-Time 
Standing 
-Full-Time Fixed 
Term 
Part-Time 
Faculty 
Graduate 
Assistants 
Other (Execs, 
Staff...) 


Figure 3. Percentage of Student Credit Hours Taught by Appointment Type: Penn 
State 

What is less obvious regarding changes in the shares of student credit hours being 
delivered by different types of faculty members is the degree to which the average tenure 
track faculty member is changing the amount of teaching being contributed. To obtain 
insight into this trend, we examined the number of student credit hours being delivered 
by tenure track faculty members relative to the number of tenure track faculty members at 
Penn State, over time. In the fall of 1992, the average number of student credit hours per 
tenure track faculty member was 208. This ratio has decreased steadily to 191 student 
credit hours in 1999 and to 157 student credit hours in the fall of 2007. The 
corresponding figure for the average full-time fixed term faculty member in the fall of 
2007 was 358 credit hours. 

Given these differences in the nature of the work being performed, it is not 
surprising to find differences in compensation. Salaries for tenure track faculty tend to be 
higher than for their non-tenure track counterparts and the differences are driven largely 
by market forces. Research talent, particularly top research talent that generates external 
grants and contracts, is in limited supply in most fields and research universities are 
willing to pay a premium for it. Ehrenberg and Zhang (2004) conducted an econometric 
analysis for a sample of four- year institutions to estimate demand functions for tenure 
track versus contingent faculty. Their analysis of salary differences revealed that the 
average salary of lecturers (a common academic title for contingent faculty) compared to 
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the average salary of all professorial faculty members at four- year colleges and 
universities in the United States declined from .642 to .607 between 1989 and 1997. 
They attribute these declining relative salaries to the large supply of Ph.D.’s seeking 
work in higher education. They see the growing salary gap between contingent and 
tenure track faculty as one of the forces prompting efforts to unionize full-time 
contingent faculty members (Ehrenberg and Zhang, 2004). 

At Penn State, the median salary for a tenure track faculty member at the 
University Park campus in the fall of 2007 was approximately $89,800. The 
corresponding figure for a full-time fixed term faculty member was approximately 
$45,400. Thus the ratio (using medians) at Penn State was about .51 in the most recent 
period. 


Economic Pressures 

Research universities pursue ambitious and seemingly open-ended missions 
through which they simultaneously aspire to generate ever larger amounts of new and 
highly valuable knowledge along with meeting their responsibility to meaningfully 
extend existing knowledge to variously defined constituencies. Public research 
universities have additional responsibilities for serving the immediate as well as longer 
term needs of taxpayers who help cover costs. In some states, student population growth 
is robust, adding further pressures on public universities. Budgets for these institutions 
vary substantially but are always finite, and basic economics teaches that open-ended 
ambitions in the face of finite resources give rise to fiscal stress. 

While feelings of fiscal stress may be widespread within higher education, the 
precise magnitude and nature of the stress varies considerably across institutions. We 
single out three dimensions (there are no doubt more) of fiscally related pressures: a) 
declining state general appropriations support; b) growing gaps in endowment income; 
and c) transformations in the research economy. 

Declining State Support 

There is great variation in the level and nature of general appropriation state 
support coming to research universities in the United States. At one extreme, there are 
states such as Florida where state support in per capita terms historically has been 
comparatively high, although Florida in the past several years has been making 
significant reductions. At the other extreme are states such as Pennsylvania where there 
is a history of low levels of per capita support for public higher education. What is more 
generally true is that a significant downturn has taken place during the past 10 years in 
the percentage of state support to public research university funding that is contributed by 
state appropriations. Regardless of whether state support historically has been high or 
low, it is in relative decline. 

Figure 4 dramatically illustrates how this decline has affected Penn State. At this 
university, the cumulative long-term impact of the slowly growing (in nominal dollars) 
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state appropriation, which translates into a decline as a percentage of the University’s 
total budget, has fundamentally altered the economic environment in which the institution 
operates. 



Tuition & Fees 


Appropriation 


1971-72 1977-78 1983-84 1989-90 1995-96 2002-03 2007-08 

Figure 4. State Appropriation vs. Tuition & Fees as a Percentage of Penn State’s 
General Funds Budget 4 


Four decades ago, state appropriations provided for roughly 70% of the 
University’s general funds operations. Today, state appropriations account for only 22% 
of the general funds budget and less than 10% of its total budget. Figure 4 shows the 
extent to which Penn State has turned to tuition as its main source of general funds. 

Penn State has been able to rely more heavily on tuition revenues as a means of 
offsetting reductions in state support. Other public research universities in the United 
States have less independent authority over the setting of tuition. Nevertheless, national 
data reveal similar shifts toward greater reliance on tuition revenues for the sector as a 
whole. 


The State Higher Education Executive Officers (SHEEO) organization has 
compiled and published the relevant data gathered from sources including the National 
Center for Education Statistics, the Bureau of Economic Analysis, the Bureau of Labor 
Statistics, and the U.S. Census Bureau. According to the 2007 SHEEO report, state and 
local higher education appropriations nationally fell from 78% to 64% of total 
educational revenue from 1982 to 2007. During that same 25 -year period, net tuition 
revenues rose from 22% to 36% of total educational revenue. In other words, the data 
represented by what might be called the “X-graph” in Figure 4 are unique to Penn State, 
but nationally higher education is similarly relying more upon tuition and less upon state 
appropriations. 


4 General funds are dollars used for educational and general purposes; the other two large budget 
components are restricted funds which support contract research, and auxiliary enterprises which include 
break-even operations such as housing and food services. 


72% 
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Research universities face market realities which translate into worries over 
competition from less expensive alternatives like community colleges. They must 
credibly make the case to prospective students and their families that the additional costs 
associated with their programs are worth bearing. 

Growing Differences in Endowments 

Financial pressures also arise from the large and growing differences in levels of 
endowment income available to research universities in the United States. Data 
describing the rank order distribution of endowments, shown in Table 1, provide one 
clear illustration of how differentially wealth is distributed across United States 
universities and support the proposition that major research universities face 
fundamentally different resource realities. Even for the 30 most well-endowed 
institutions in the nation, the differences are striking. Endowment values tail off very 
quickly from #1 Harvard’s $34.6 billion down to #30 University of North Carolina’s $2.2 
billion. (Penn State ranked #46 in 2007 with an endowment valued at $1.6 billion.) One 
way to appreciate this profound difference is to realize that most universities in the 
United States have total endowments smaller than the annual amount realized in the 
growth of the Harvard or Yale or Stanford endowment in a typical year. 

It is also the case that the great disparity in wealth is growing; in other words, 
each year it falls off more and more quickly from the top. Even among the top 30 
institutions, the endowment gap - the differences between the elite few and the rest of 
colleges and universities - is visibly widening. In 2007, Harvard’s endowment was 16 
times greater than that of #30 North Carolina’s. Ten years ago (that is, using 1997 
endowment values), Harvard was also #1, but its $10.9 billion endowment was 12 times 
the value of then #30 Purdue’s endowment of $856,000. If these comparisons are 
extended to a broader group, by comparing #1 to #100, the widening gap is equally 
apparent. In 2007, the largest endowment was 48 times the value of the #100 endowment, 
up from 36 times larger in 1997 (The Chronicle of Higher Education , 2008; National 
Association of College and University Business Officers, 2008). 

The widening differences in levels of wealth need to be viewed in light of these 
institutions’ common commitment to the conduct of cutting-edge research. Universities 
with endowments in the range of one to two billion dollars (such as Penn State) are 
competing head-to-head with universities in the $10+ billion range (such as MIT), and 
this sets in motion powerful forces to enhance the competitiveness, through alternative 
means, of less wealthy institutions. 
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Table 1. Fiscal Year 2007 Endowment Values and Percentage Change from 2006 


1. Harvard 

2. Yale 

3. Stanford 

4. Princeton 

5. Texas 

6. M.I.T. 

7. Columbia 

8. Michigan 

9. Pennsylvania 

10. Texas A&M 

11. Northwestern 

12. California 

13. Chicago 

14. Notre Dame 

15. Duke 

16. Washington University (St. Louis) 

17. Emory 

18. Cornell 

19. Rice 

20. Virginia 

21. Dartmouth 

22. Southern California 

23. Vanderbilt 

24. Minnesota 

25. Johns Hopkins 

26. Brown 

27. Ohio State 

28. Pittsburgh 

29. University of Washington 

30. North Carolina - Chapel Hill 


2007 Value 

Percent Change 

$34.6 

19.8% 

$22.5 

25.0% 

$17.2 

21.9% 

$15.8 

21.0% 

$15.6 

18.0% 

$10.0 

19.3% 

$7.1 

20.4% 

$7.1 

25.4% 

$6.6 

24.9% 

$6.6 

16.8% 

$6.6 

26.5% 

$6.4 

16.2% 

$6.2 

27.5% 

$6.0 

34.7% 

$5.9 

31.4% 

$5.6 

18.9% 

$5.6 

14.2% 

$5.4 

25.5% 

$4.7 

17.1% 

$4.4 

20.8% 

$3.8 

21.6% 

$3.7 

21.2% 

$3.5 

18.4% 

$2.8 

26.1% 

$2.8 

19.1% 

$2.8 

21.4% 

$2.3 

17.1% 

$2.3 

25.0% 

$2.2 

21.7% 

$2.2 

32.1% 


Source: National Association of College and University Business Officers, 2008. 


Transformations in the Research Economy 

An analysis recently completed by Roger Geiger (based in part upon his analysis 
of changes in teaching, research, and spending at 33 private and 66 public universities 
from 1980 through 2000) has led him to conclude that the expansion of academic 
research since the late 1990s has driven, and continues to drive, a substantive and 
transformational restructuring of faculty work (Geiger, 2008). 

Geiger’s case can be summarized in terms of four factors. First, a consensus has 
emerged that the contribution of university research to economic development is crucial 
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to the competitiveness of the U.S. economy, and federal support policies have strongly 
encouraged (directly or indirectly) an expectation that faculty work will contribute to 
economic development through research. Second, while interdisciplinarity has long been 
in fashion for teaching and research, the practical impact in terms of hiring and funding 
decisions have been vastly weighted toward the research side. Third, Geiger has 
documented the proliferation of university centers and institutes whose essential function 
is to facilitate research. He notes that they do this both with their own core faculty and 
through the provision of research support to other faculty members, to the extent that 
increasingly, research university faculty define themselves by membership in 
departments and by affiliation with one or more research units. Fourth, Geiger notes that 
the bar of research accomplishment has been raised for new Ph.D.’s seeking to obtain 
academic appointments, and for subsequent promotion and tenure. 

The argument is, in short, that perceptions of excellence and the responsibilities of 
regular faculty are becoming more research-oriented, and the function of teaching is 
increasingly shifted to non-tenure track faculty members. We find these arguments 
compelling, and we are persuaded by Geiger’s conclusion that at this time - early in the 
21 st century - higher education is likely at or near a tipping point, in which a 
transformation is occurring in the roles, responsibilities, and structure of the faculty for 
research universities. 


Responses and Consequences 

We have outlined several contextual pressures that we believe have encouraged 
research universities to respond by substantially increasing reliance on non-tenure track 
members of the faculty. At first glance, this might appear to be a straightforward matter 
of cost-cutting. We have seen that non-tenure track faculty members are less expensive 
to hire and that they generate more student credit hours per capita than is the case for 
their tenure track colleagues. Indeed, our first approximation is that non-tenure track 
faculty generate about twice as many student credit hours at approximately half the 
salary. 


However, the underlying economic forces are more complicated than this, since 
these institutions are also aggressively seeking external grant and contract support for 
their research missions. It is the tenure track faculty that has explicit responsibility for 
leading research efforts, and it could be very shortsighted indeed for a research university 
to pull away substantially from its investment in tenure track faculty members. 

As a consequence, decision makers at research universities are being pulled in 
different directions, and delicate and difficult balances must be reached. We suspect 
great variation exists across the disciplines in how these balances are struck and that 
much will depend on factors such as the degree to which external funding opportunities 
exist for a particular unit, the unit’s history of success at competing for external funds, 
and judgments about future prospects for success. 
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We looked more carefully at the Penn State data in an effort to ascertain the 
degree to which this University has substituted tenure track for non-tenure track faculty 
appointments. We found that even though the percentage of faculty appointments on the 
tenure track has been dropping, the absolute numbers of tenure track faculty members has 
increased, albeit modestly. Between the fall of 1998 and the fall of 2007, the number of 
tenure track appointments at Penn State (University Park) increased from 1,612 to 1,646 
or 2.1%. It follows that Penn State has not back-tracked on its investment in tenure track 
faculty positions, presumably because of its commitment to be competitive for grants and 
contracts that are well aligned with the faculty’s intellectual interests. It is nevertheless 
true that much greater growth has taken place at Penn State with respect to its fixed term 
faculty appointments which grew from 609 to 999 or 64% during the same period. 

We also divided the category of fixed term appointments into its two components, 
multi-year fixed term and single-year fixed term, and found a substantial difference in the 
growth rates. In particular, between 1998 and 2007 the multi-year fixed term group grew 
from 98 to 285 (191.8%) while the single-year fixed term group grew from 511 to 714 
(39.7%). Penn State is more heavily invested in the single-year fixed term type of 
appointment, but is shifting in the direction of greater emphasis on multi-year fixed term 
appointments. 

Educational Consequences 

Should we be concerned about the impact of this shifting mix of faculty 
appointments on the quality of the education being delivered by these institutions? There 
have been worries about the erosion of academic freedom as institutions rely more 
heavily on faculty with fixed term contracts. Questions have also been asked about the 
impact on the quality of teaching. 

Concerns about the status, vitality, and academic freedom of the professoriate 
have a notable history in American higher education. Efforts to protect the professoriate 
led to several versions and interpretations of the American Association of University 
Professors (AAUP) and the American Council on Education’s 1940 Statement of 
Principles on Academic Tenure. That statement, which has been endorsed by over 180 
professional and scholarly groups over the past seven decades, reflects the high value that 
academe has traditionally placed on a secure and stable environment in which faculty can 
work. The AAUP has historically called for all full-time faculty appointments to be 
tenure track, except for special appointments clearly designed as short-term 
arrangements. In general, substantial reliance on large numbers of non-tenure track 
faculty is perceived as a threat to norms such as academic freedom (Day, 2004). 

It would be simplistic to assert that X percentage of teaching by non-tenure track 
faculty is acceptable, while X+l percentage is harmful. However, there is a sense that 
there is some line, even if ill-defined, about which institutions should be cautious. In 
1993, the AAUP recommended limiting the use of special appointments for part-time and 
non-tenure track faculty to no more than 15% of the faculty, with responsibility for not 
more than 35% of instruction within any given department. Similarly, the Carnegie 
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Foundation for the Advancement of Teaching has suggested that part-time faculty be 
limited to only 20 percent of undergraduate instruction (Balch, 1999). 

There is evidence that when part-time faculty replace full-time faculty there are 
detrimental effects on students and the institution (Benjamin, 1998; Gappa, 2000). 
However, there is also evidence showing that when properly utilized and supported, part- 
time and full-time non-tenure track faculty members can be extremely effective teachers 
and valuable resources. Indeed, students can benefit substantially from interactions with 
part-time faculty who bring practical, workplace skills to a college or university. Gappa 
and Leslie (1997) have argued that part-time faculty members are “professionally 
qualified for the work they do” (p. 12) and that in general, there are more similarities than 
differences between part-time and full-time faculty. 

Of course, there are concerns besides possible positive or negative repercussions 
on the quality of teaching and learning. For example, on the positive side, compelling 
reasons can be offered for relying on clinical faculty and professors of practice (who are 
usually not on tenure track appointments), who can share valuable field experience that is 
especially needed in applied fields. On the negative side, non-tenure track appointments 
are often perceived as being a threat to academic freedom; as an attenuation of faculty 
participation in institutional governance; and as a mistreatment of the individuals in terms 
of job security, satisfaction, salary, benefits, and working conditions. From a 
programmatic perspective, there are both advantages and disadvantages to the use of 
contingent faculty. 

We noted earlier that groups such as the AAUP and the Carnegie Foundation have 
attempted to promulgate guidelines for the utilization of part-time and contingent faculty. 
However, these statements have really not been embraced by research universities. In 
2007, we conducted a survey of Association of American Universities member 
institutions and examined whether and how ideas about the desirable proportion of 
teaching by non-tenure track faculty are translated into practice. We found that only 1 of 
24 responding universities have a formal policy at the university level on instructional 
delivery by faculty appointment type (although 10 of 24 reported policies at the college 
or department level). 


Looming Challenges 

Recent turmoil in U.S. and international economies will undoubtedly affect the 
fiscal stresses on public universities described above. We suspect that, if anything, those 
economic pressures will intensify. We highlight two more factors that may create 
additional challenges for public research universities in the management of their faculty 
resources in the next few years: the graying faculty and a growing public-private salary 
disparity in the race for top scholars. 
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The Graying Faculty 

TIAA-CREF data (Conley, 2008) show that the average age of full-time faculty 
members has increased from 47 to 50 in the last twenty years and that the bulk of older 
faculty are still in the pipeline. TIAA-CREF research (Yakoboski, 2007) also finds that 
although faculty members tend to retire at ages slightly older than the general population 
(about age 66 compared to 62) and that some professors would like to stay as long as 
possible (37% plan to work to age 70 or beyond), an unprecedented wave of retirements 
is coming to higher education in the near future. 

Just as the graying of tenure track faculty is a national phenomenon, it is readily 
observable at Penn State as well. Since at least the early 1980s, the percentage of full- 
time faculty age 60 or older at Penn State was very steady at 8% to 10% until it inched 
above 10% in 2000 and has increased steadily upward to approximately 15% in 2007. 

The rising percentages of older faculty are significant because they make it clear 
that major changes will be occurring over the next 5-10 years in the composition of the 
faculty in U.S. higher education. In addition, at Penn State we have also observed 
differences in the turnover rates by type of faculty appointment. The normal turnover of 
tenure track faculty at Penn State is about 4% per year for tenured associate professors, 
5% for tenured professors, and about 9% for untenured assistant professors on the tenure 
track. For faculty on fixed term contracts, the turnover rate is 15% per year. Thus, the 
shift toward relying more heavily on fixed term faculty accelerates the rate at which 
further change can occur. 

Growing Public-Private Salary Disparities 

We highlight one more complicating factor that perhaps is not as well 
recognized as it should be: namely, the diminishing capacity of many public colleges and 
universities, including prominent research universities such as Penn State, to successfully 
compete in the evolving labor market for top scholars. 

Tracking faculty salaries for 214 doctoral and research universities from the 
1970s through the 1990s, Alexander noted that, “...data show that from 1970 to 1980, 
faculty salary disparities between public and private universities only slightly advantaged 
private university faculty,” but followed distinctly different trajectories in the 1980s and 
1990s (Alexander, 2000, p. 3). His analysis showed that the public-private gaps at each 
rank for various institutional categories went from practically negligible in 1979-1980 to 
what he termed as an emerging “silent crisis” by 1997-98. For example, for the 
universities in Alexander’s dataset, in 1979-80 average salaries for full professors were 
only 2% higher ($69,000 versus $67,700) at private research universities; by 1997-98, 
average salaries for full professors at those private universities were 30% higher ($96,000 
versus $74,000) than for their counterparts in public universities. 

Recently published AAUP data show average salaries for professors at private 
doctoral universities at $144,256 in 2007-08; that figure compares to $109,569 for 
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professors at public doctoral universities, for a private university advantage of 32%. The 
private doctoral universities’ 2007-08 advantage was 20% for associate professors and 
21% for assistant professors (American Association of University Professors, 2008). 

This is important because, in fact, public research universities such as Penn State do 
compete for nationally and internationally recognized scholars in many fields. In 2007, 
Penn State’s provost’s office compiled a listing of the university’s academic programs or 
major subfields - undergraduate, graduate, and professional - that are considered to be 
among the top 20 in the nation by at least one generally recognized or reasonably 
reputable source. Some of these are programs ranked by U.S. News & World Report , 
others by the National Research Council, others by National Institutes of Health research 
funding, and others by scholarly or professional societies. A few more - lacking a formal 
ranking mechanism in the discipline - would undoubtedly be included on any such list by 
knowledgeable academic or professional leaders of the field. That exercise showed that 
there are currently more than 90 disciplinary programs or major subfields at Penn State 
that would rank among the top 20 programs nationally, and more than 50 of those that 
would fall into the top 10. The point is that Penn State and other public research 
universities generally stack up well with the best private research universities; they are 
nationally and internationally competitive in terms of program quality and prestige. 

Programs earn a reputation for high quality through a combination of 
excellence in teaching, research, and visibility within the profession, but the single key 
ingredient is the strength, scholarly productivity, and recognized accomplishment of 
faculty. Thus, as public universities such as Penn State strive to succeed in the arena of 
first-class research institutions, they have no choice but to compete in the national and 
international market for first-class scholars. 

This is not empty rhetoric. Because of the importance of the ongoing effort to 
recruit and retain top academics, in 2001 Penn State’s provost and institutional research 
office began collecting information from the deans of cases in which they worked to keep 
faculty members in the face of offers from other universities. We know, therefore, that 
from 2001 through 2008, Penn State either lost faculty members to, or successfully 
negotiated against, every other Big Ten university (Wisconsin, Michigan, Ohio State, 
Northwestern, and so on); Ivy League universities including Brown, Columbia, Cornell, 
Harvard, Penn, and Yale; and many other premier U.S. and international universities, 
such as the University of Chicago, the University of Auckland, the Stowers Institute for 
Medical Research, McMaster University, University of California, Carnegie Mellon, 
Stanford, Tulane, Vanderbilt, and the University of Virginia. In short, the competition for 
top scholars is real. It is serious, ongoing, and expensive - and likely to become stiffer for 
ambitious public research universities, in comparison to their private university peers, in 
the near future. 


Conclusions and Implications for Practice 

It is clear from both national and individual university data that research 
universities have experienced a significant weakening of state support for their core 
academic enterprises over the past several decades. The erosion of public financial 
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support has pushed public universities to turn increasingly to student tuition and fees to 
offset these declines. For many of these universities, growth in tuition and fees has also 
been limited by their commitments to serve students from a wide range of socio- 
economic backgrounds (and ability to pay) as well as by legislative and other constraints 
on their ability to increase tuition. Many public research universities are also pressured 
to open their doors to additional students in states where populations are growing rapidly. 

In addition, endowment incomes are comparatively modest in most research 
universities and have not kept pace with the growth realized by universities with the 
largest endowments. Moreover, most public research universities have only recently 
become engaged in raising significant amounts of philanthropic support relative to their 
private university peers. The result is a growing gap in institutional wealth (endowment) 
among research universities in an increasingly competitive environment for talented 
students and faculty, extramural research funding, and national and international prestige. 

Research universities caught in an ever- tightening vise of rising costs and 
increasingly scarce resources have by necessity turned to a variety of strategies to 
enhance revenues and reduce expenses. One of the most significant responses has been 
the steady movement toward greater utilization of contingent faculty at the relative 
expense of tenure track appointments. This shift appears to have been evolutionary in 
most cases and carried out within colleges and departments, where most academic 
staffing decisions are made, rather than being the result of a deliberate, centrally 
orchestrated university-level strategy. It is probably better characterized as an 
incremental, “muddling- through” approach as academic units adjust to fiscal constraints 
while optimistically hoping or planning that the resource situation will improve in the 
future. 


What we may be seeing then is movement away from the conventional idea of a 
tenure track faculty member who excels as a “triple threat” in teaching, research, and 
service and toward a more specialized approach or division of faculty labor. Our data 
indicate that tenure track faculty are teaching a declining share of student credit hours, 
which is entirely logical as tenure track faculty represent a declining proportion of all 
faculty and the pressures to produce scholarship, increase extramural funding, and 
participate in outreach activities are heightened. The increase in research activity among 
research universities in recent decades has been remarkable, and while these resources 
have released mainly tenure track faculty from some teaching responsibilities and freed 
up resources to use for alternative instructional personnel, support for the research 
enterprise in terms of facilities, equipment, and unrecovered overhead have added further 
stress to many department, college, and university budgets. 

The use of contingent faculty has therefore become one of the dominating 
national trends of recent decades, having shifted from a situation where nearly all full- 
time faculty appointments were tenure-eligible to the current pattern where a majority of 
the new appointments are contingent. However, there is a range of contingent 
appointments that has potentially important implications for higher education and the 
professoriate. Some contingent appointments may be part-time and involve teaching 
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regular or occasionally offered courses. Others may be full-time, extended one year at 
the time, while still others may involve longer-term commitments up to five years. Full- 
time fixed term faculty at research universities, particularly those on multi-year 
appointments, may be required (or at least encouraged) to engage in some level of 
scholarly research, although not at the same level as tenure track faculty given the 
differences in the teaching expectations. 

Multi-year fixed-term faculty appointments are a kind of hybrid that we predict 
will become increasingly important in the future research university as fiscal stresses 
become no less severe in coming years. Growing specialization of faculty roles and 
associated workloads may result in those faculty members on the tenure track becoming 
more research oriented while those on fixed-term contracts emerge as a new class of 
teaching faculty. Those on multi-year appointments have considerable job security, and 
they may well become the work horse of the university in terms of instruction, at least in 
lower division teaching. In this light, we repeat our finding that the fastest growing 
contingent faculty appointment type at Penn State between 1998 and 2007 was the multi- 
year fixed term variety. Stepping away from the more complementarily view of faculty 
roles (i.e., research informs teaching and teaching informs research) has many 
implications for research universities in terms of the educational process and what have 
traditionally been a major differentiating element of comprehensive research universities. 

We are not yet in a position to develop tight arguments about what combination 
and sequence of coping strategies is most likely to occur under what set of circumstance, 
but we hope to learn more as we gather and analyze additional data. It is likely to be the 
case that there are few, if any, coping strategies that come without costs or associated 
consequences. However, we offer the following four recommendations for how a 
university may strategically support excellence through the hiring, development, and 
utilization of outstanding faculty. 

The first is to use cluster hiring to attract outstanding faculty and build excellent 
programs. There is ample evidence that cluster hiring in specific focused areas works 
well in establishing discipline leadership and critical mass. The same approach can be 
effective in creating or expanding existing interdisciplinary thrusts, particularly when 
institutes help by supporting co-hires between colleges. 

The second is to provide more effective review of and support for fixed-term 
faculty. Because building excellent programs and departments requires major 
contributions from non-tenure track faculty, it is important to ensure that fixed-term 
faculty can find job satisfaction, and be able to see a career path that encourages them to 
stay and develop excellence. Units should encourage and (to the extent feasible) support 
professional development for fixed-term faculty through such means as attendance at 
regional or national professional conferences. 

The third idea is to strengthen faculty mentoring, support, and development. 
Every hiring decision represents a major commitment of resources and potentially the 
high costs of a poor initial choice, or of not giving a new faculty member the best chance 
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to succeed. Appropriate mentoring by senior faculty and flexibility along the career path 
to accommodate major life course developments such as childbirth, adoption and family 
illness is part of the way to support faculty and staff. College and departmental support 
should include (again, to the extent feasible) encouragement and funding for professional 
travel and development as well as consideration of the need for work-family balance. 

An ongoing challenge is to recruit and retain women and minority faculty members in 
greater numbers, especially in the STEM (science, technology, engineering, and 
mathematics) disciplines, and to ensure the development of the capabilities of these 
faculty members. All faculty need to be supported and integrated into the lives of their 
colleges and departments. Given the large and growing role of non-tenured faculty, 
universities probably should be paying more attention and asking whether they are truly 
employing the best practices available to support all faculty members. 

Fourth, planning and institutional research practitioners can and should contribute 
as well. They can develop relevant data and analyses, and share that information as 
widely as possible among academic leaders: provosts, chancellors, vice provosts, deans, 
department heads, faculty senates, and others involved in academic governance and 
management. The issues are serious and timely and surely the problems will not be 
solved by data alone, but good institutional research can raise awareness, clarify the 
understanding of challenges, and help to inform smart planning and decision making. 
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Coffee & Conversation: Gathering Information from our Graduates 
Peter Feigenbaum, Fordham University 

This Coffee & Conversation session focused on the current and potential uses of 
Graduate Placement surveys. The audience was invited to discuss the proposal that 
Graduate Placement surveys are the ideal venue for collecting valuable outcome 
measures of an institution's effectiveness in fulfilling its mission. 

Graduate Placement surveys: 

I. What we do now. 

The Graduate Placement surveys that I examined from a selected group of institutions 
tend to be very similar in content. For example, the surveys from UPenn, Notre Dame, 
and Boston College ask about similar content areas: job placement, graduate school 
attendance, and “other” activities (e.g., community service, military service, and travel). 
More variable are the form of the surveys (e.g., paper-and-pencil vs. online) and the 
conditions and timing of the administration of the surveys (e.g., before graduation, 3 
months out, six months out, etc.). 

The main focus of Graduate Placement surveys is on jobs and salaries — as the name 
suggests. The job questions tend to be fairly standard, and include length of employment, 
employer, and the relation of job to college major. The salary questions are a bit more 
varied, owing to the sensitivity and confidentiality of the information requested. Some 
institutions ask for exact figures, while others only ask graduates to identify a salary 
range. Some surveys also ask about start-up and annual bonuses. 

II. Why we do it. 

The pressure to demonstrate the employability of our graduates is coming from many 
sources, particularly market forces. College guides, administrators, parents, and the 
students themselves are all concerned with employment outcomes, and some of our IR 
colleagues have reported that IPEDS may soon require institutions to provide this 
information. It is widely acknowledged, however, that these data are very difficult to 
obtain, as evidenced by the generally low survey response rates from students once they 
have graduated and left their respective institutions. 

III. What we could do. 

Despite the logistical problems, there is much to be gained from surveying our graduates 
3, 4, 5 years out and more. We could be assessing our institutional effectiveness and our 
mission. Graduate Placement surveys are the ideal venue for collecting valuable outcome 
measures of an institution's effectiveness in fulfilling its mission. Among the goals of 
such an outcome assessment would be an answer to the question: Have we produced the 
kind of graduates that we promised to produce in our mission statement? 

In the case of Jesuit institutions, such as Fordham University, we would do well to ask: 
Are we producing graduates who value learning and wisdom, who are committed to 
serving others, and who are dedicated to lifting the poor out of poverty? 
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How can we know these outcomes? I suggest that the answer is by surveying our 
graduates two-, three-, five-years out, and longer! By asking them more nuanced 
questions than we do in the typical Graduate Placement survey. For example, of the pre- 
med students who graduate from Fordham, we might ask whether they volunteer their 
services to charitable organizations (such as Doctors Without Borders) in addition to their 
regular clinical activity. Similarly, of the pre-law students who have graduated and begun 
their law careers, we might ask whether they also volunteer their services pro bono to 
work as Legal Aid attorneys. And of the Accounting majors who have graduated, we 
might ask if they also volunteer their services at tax clinics for poor people. 

IV. Why we should do it. 

The main reason for suggesting that Graduate Placement surveys might be extended and 
developed into “Mission Critical” surveys is that they would enable us to obtain direct 
measures of the quality of education that we provide. Rankings in popular magazines 
tend to use indirect measures of educational quality in assessing the “business” side of 
academic institutions. But Institutional Researchers — as we learned from this morning’s 
Plenary session — have need to be proactive in defining the measures that define our 
institutions. If we wish to demonstrate that we produce graduates who reflect the values 
of our mission — such as people with ethical standards — then developing our Graduate 
Placement surveys into surveys of institutional effectiveness in the broadest sense is an 
ideal place to start. 

V. Limitations to this approach. 

I would be remiss if I did not mention some of the limitations of the concept I am 
proposing. Among the technical limitations are the problems with data collection — 
particularly the problem of low survey response rates, especially as the time frame is 
extended outward to five years and beyond. As we all know, it is difficult to obtain high 
rates of response over time. One possible antidote to this problem is to develop stronger 
bonds with our students while they are still undergraduates. The more we instill in them a 
sense of community and belonging through academic advising, social activities, and 
campus involvement, the more likely they are to stay in contact with us years later. 

Another limitation that we would want to remove is the administrative conflict among 
offices that keep contact with our graduates. It is fairly typical for IR offices and Alumni 
Affairs offices, for example, to be in conflict over surveying our graduates, with Alumni 
Affairs adopting a proprietary attitude about current addresses of graduates in order to 
protect their function of soliciting much-needed donations from alumni. There is also the 
problem of over-surveying and maintaining the good will of our graduates. These 
problems are not insurmountable, I would argue, and do not prevent us from obtaining 
information that is vital to the assessment and continued success of our institutions and to 
the execution of our mission. 
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DOES FINANCIAL AID STATUS AFFECT STUDENT PERFORMANCE, 
RETENTION, PERSISTENCE, AND ACADEMIC SUCCESS? 


Dr. Corby A. Coperthwaite 

Director of Planning, Research and Assessment, Board of Trustees, Connecticut 
Community-Technical College System 


Mr. Benjamin Klimczak 
Graduate Student 
The Pennsylvania University 

Abstract 

Does student participation in financial aid programs affect academic 
performance? The sample for this study includes all (5,144) fall 2000 first-time degree or 
certificate seeking students within the twelve Connecticut Community Colleges. Students 
were tracked for six academic years. Although some differences in performance exist 
among the various financial aid participants and non-participants, these differences 
cannot be attributed to the financial aid groups alone. Variables, both demographic and 
college specific, are interacting with each other to form significant combinations that 
allow for estimates of the likelihood of attaining known academic markers such as 
successful completion of college-level math, attaining a first semester GPA >= 2.0, being 
retained fall to spring and persisting fall to fall. 

The question now becomes why (or why not) students in various categories 
identified by logistic regression, are succeeding in the first semester, being retained and 
persisting, regardless of financial aid participation? For example, why are students in the 
highest reading group less likely to persist? Are they transferring early? In general, why 
are Hispanic males less likely to achieve a first semester GPA of >= 2.0 than other 
students? Why are Black and Hispanic students especially at three of the urban colleges 
performing so differently than other students? What other factors in addition to those 
revealed in this study, are impacting students and how? What strategies can be taken to 
further enhance the success of students? 

During the 2005-2006 award year the twelve Connecticut Community Colleges 
enrolled 64,183 credit students (annualized unduplicated headcount). There were 33,972 
financial aid applicants, 19,184 financial aid recipients, and 12,744 Pell Grant Recipients; 
approximately a 63% increase in all three categories since FY 2001. The total Pell Grant 
money awarded was $23,584,175, a 99% increase in dollars awarded since FY 2001. 

The Connecticut Community College System’s financial aid packaging policy is 
primarily need based and designed to first meet the direct costs (tuition, books and fees) 
of eligible students. There is a considerable amount of unmet need in the system, and our 
policy is to discourage the use of loans that will subject an already financially 
disadvantaged student population to increased debt. Our typical financial aid recipient in 
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FY 06 was female, 32 years old, single, head of household, with an average household 
size of 2.5 members, and a 2004 annual income of less than $24,000. 

The objective of the Pell Grant program, like that of other need-based grants, is to 
provide opportunity so that socio-economic status is not a barrier to access, retention, 
persistence, performance and overall academic success (workforce preparation, degree 
completion and/or transfer). Is the objective of the Pell Grant program being realized, or 
is there so much unmet need in the form of food, rent and utilities, for example, that 
among the poorest students socio-economic status is still a barrier to a successful higher 
education experience despite the best of financial aid programs? 

A longitudinal study of a 1996 cohort of students demonstrates that, on a national 
level, students from first generation and low-income backgrounds are less likely to enroll 
in postsecondary education and less likely to persist through graduation (P. Thayer, 

2000). Thayer found that students in the top family income quartile completed a 
baccalaureate degree at a 74% rate, as compared to 5% for those in the bottom income 
quartile. Do these same results hold at the individual colleges? 

Results from a similar study of fall 1999 first time undergraduate students at a 
four- year selective institution indicate that students receiving the largest amounts of Pell 
money outperformed their peers in terms of persistence and graduation (Z. Yang, 2006). 
Yang suggests that the contrary results found in his study may be attributable to 
“restricted range” issues. A national sample should have much more variance in 
academic preparedness than that taken from a selective university, where the level of 
academic preparedness among the student body should be higher based upon admissions 
requirements. 

If Yang is correct, then two-year colleges with open admissions should be more 
likely to mirror Thayer’s national sample, for which academic preparedness is a more 
important predictor of academic success than socio-economic status. Support for this 
premise is provided by other research, with findings suggesting that on a national level at 
two-year colleges, Pell recipients often have many of the recognized risk factors for 
leaving college early, such as not graduating from high school or finishing with a GED or 
equivalency certificate; delaying enrollment in postsecondary education; being 
financially independent; having dependents other than one’s spouse; and being a single 
parent. At first pass in the analysis, it does appear that Pell recipients are less likely to 
persist than their non-Pell recipient counterparts. However, once variables such as high 
school curriculum rigor and achievement test scores are controlled for, no differences in 
persistence are found (Wie & Horn, 2002). In fact, although financial difficulties are 
often cited as one of the top three reasons why students leave college, some would argue 
that students tend to overstate their financial difficulties. Financial difficulties are seen as 
a more acceptable reason for leaving than alternative explanations such as motivation, 
academic difficulty, etc (D. Masursky, 1977). In sharp contrast are research findings that 
suggest college costs, especially unmet need for living costs and loans, have a substantial 
and direct influence on persistence (St. John & Starkey, 1996; M. Wang, 2006). 
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Other variables often thought to combine with socioeconomic status to affect 
student persistence are mother’s education, “first generation” status and ethnicity. 

Ishitani & DesJardinsm, (2002) provide evidence that students whose mothers graduated 
from college are less likely to leave than other students; this effect is especially strong in 
the second year of college. 

The impact of “first generation” status may be more myth than reality. Research 
has shown that “first generation” students are more likely to enroll in developmental 
education courses; but upon further analysis, income, high school GPA and age, not 
“first- generation” status, are more important predictors of academic success. Further, 
“first generation” status has little or no impact upon GPA (Brown & Burkhardt, 1999). 

A recent report compared the outcomes for two cohorts of students, those who 
began college at two-year institutions with the goal of attaining a degree in 1989, and 
those who began in 1995. African-American students in the 1995 cohort were less likely 
to earn a degree (26.5%) than those in the 1989 cohort (31.5%). The gap was even larger 
for Hispanic students: 54.9% for the 1989 cohort compared with 39.6% for the 1990 
cohort (Cook & Cordova, 2006). There was no accounting for socio-economic status in 
this report. Although the gaps are a reality, and in the opposite direction of what one 
might expect given all the efforts in recent years to bridge these gaps, the question of why 
remains largely unanswered. 

Overall the results of research investigating the impact of financial aid on 
educational outcomes are mixed and therefore inconclusive. Some of the research 
provides for the positive impact of receiving financial aid, and other research shows a 
minimal effect that is mitigated by a host of other variables. Results of research on a 
national level are not always congruent with results revealed at individual college 
campuses. There is also evidence that the impact of financial aid is greater for students at 
two-year colleges than for those attending four- year institutions (Pascarella & Terenzini, 
1991). Pascarella and Terenzini (1991) report that the preponderance of the evidence 
indicates that after controlling for academic ability, students who receive financial aid are 
as likely to persist in college as those who do not receive such aid, and further suggest 
that receiving financial aid may be compensating well for the negative impact of a low 
socio-economic background. 

The purpose of this study is to clear up some of the ambiguity by quantifying the 
impact of financial aid on student retention, persistence, performance, and academic 
success. The study attempts to address eight research areas. 

(1) Over a six year period, how many students in the various financial aid categories 
attain positive outcomes, and do differences exist among the groups? Positive outcomes 
include: 

• award of a certificate without transfer 

• award of an associate degree without transfer 

• award of a certificate and transfer 
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• award of an associate degree and transfer; 

• transfer with no award; and 

• still enrolled without an award, having earned 30 or more credits. 

(2) Over a six year period, how many students in the various financial aid categories 
attain the positive marker of academic progress: successful completion of College -Level 
Mathematics? To what extent and in what manner do students differ within and among 
the various financial aid groups with respect to attainment of this marker? 

(3) How many students in the various financial aid categories attain a positive marker of 
academic progress: a first semester GPA of 2.0 or higher? To what extent and in what 
manner do students differ within and among the various financial aid groups with respect 
to attainment of this marker? 

(4) What are the strongest predictors of first semester GPA among the demographic 
variables: age, gender and ethnic group; and institutional variables: reading group, 
financial aid group, full-time part-time status, retention (fall to spring), and persistence 
(fall to fall)? 

(5) How many students in the various financial aid categories attain a positive marker of 
academic progress: retention (fall to spring)? To what extent and in what manner do 
students differ within and among the various financial aid groups with respect to this 
marker? 

(6) What are the strongest predictors of retention (fall to spring) among demographic 
variables: age, gender and ethnic group; and institutional variables: reading group, 
financial aid group, full-time part-time status, and first semester GPA? 

(7) How many students in the various financial aid categories attain a positive marker of 
academic progress: persistence (fall to fall)? To what extent and in what manner do 
students differ within and among the various financial aid groups with respect to 
attainment of this marker? 

(8) What are the strongest predictors of persistence (fall to fall) among demographic 
variables: age, gender and ethnic group; and institutional variables: reading group, 
financial aid group, full-time part-time status, first semester GPA, and retention (fall to 
spring)? 


Method 


Participants 

The sample for this study includes all fall 2000 first-time degree or certificate 
seeking students. In Banner terms this means students for whom “student type = N”; 
“degree <> 000000”; “first credit semester = summer 2000 or fall 2000”, “cumulative 
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hours earned = 0 at the start of the first credit semester”, and “cumulative transfer hours 
earned = 0 at the start of the first credit semester.” Banner provides student demographic 
variables that include ethnicity, gender, and age. Age is recoded into three groups >= 46, 
22 - 45, and <22. Banner also provides college related variables including first semester 
GPA, reading scores, financial aid information, first semester FT/PT attendance, program 
of study, and first semester college of record. GPA is recoded into two groups: GPA of < 
2.0 and GPA >= 2.0. 

A student’s reading score serves as a proxy variable for academic ability. There 
are not enough Sentence Skills and Mathematics test scores available for these to be 
useful in this analysis. For those students with valid reading scores (4,204), the scores 
are segmented into four quartiles or ability groups. Reading scores can range from zero 
to 120. For this particular group of students, the minimum reading score is one, the 
maximum is 120, the mean is 73.3042, and the standard deviation is 21.67531. For this 
sample, the upper limit of each quartile is 58.76, 73.3, 87.8 and 120 respectively. There 
are 940 students for whom no reading score is available. 

Students are assigned to four financial aid groups: (1) Pell Grant Only, (2) Non- 
Pell Aid, (3) Pell Grant and Other Aid, plus (4) No financial aid. Program of study is 
recoded into two groups: occupational or liberal arts/general studies. 

The sample includes 5,144 students: 3,010 Caucasian, non-Hispanic students; 853 
Black, non-Hispanic students; 769 Hispanic students; 128 Asian or Pacific Islander 
students; 37 non-Resident Alien students; 21 American Indian or Alaskan Native 
students; and 326 students whose category is not known. There are 2,831 females and 
2,313 males in the sample. The age of students in the sample ranges from 13 to 74, for an 
average age of 22; there are five students in the sample whose age is not known. There 
are 1,185 students with a GPA of < 2.0; 3,093 with a GPA >= 2.0; and 236 students with 
no GPA recorded. Of these students 2,692 attended full-time, 2,433 became Liberal Arts 
and General Studies majors, and 2,711 students enrolled in occupational programs. 


Design and Procedure 

Chi Square 

To address the first research question, students are followed over a six year period 
(fall 2000 through spring 2006) and the number of positive academic outcomes for 
students in the various financial aid groups were counted. Chi Squares were computed to 
determine any significant differences among the various financial aid groups. 

Analysis of Variance 

The second research question is addressed by again following students over a six 
year period (fall 2000 through spring 2006) to determine the manner and extent of their 
progression through College-level Mathematics. The analysis begins with where students 
start in mathematics (Pre- Algebra, Elementary Algebra or College-Level Math) rather 
than the perhaps more important variable of where students place; placement data for this 
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group of students are incomplete and unreliable. Analysis of variance is used to identify 
statistically significant differences in student progression through college-level math 
based on where students start and their financial aid grouping. 

The third research question is addressed using analysis of variance to identify 
statistically significant differences among the various financial aid groups with respect to 
first semester GPA, after controlling for academic ability (reading score). 

The fifth research question is addressed using analysis of variance to identify 
statistically significant differences among the various financial aid groups with respect to 
retention (fall to spring), after controlling for academic ability (reading score). 

The seventh research question is addressed using analysis of variance to identify 
statistically significant differences among the various financial aid groups with respect to 
persistence (fall to spring), after controlling for academic ability (reading score). 

Binary Logistic Regression 

The fourth, sixth and eighth research questions are addressed by means of multi- 
step binary logistical regressions. The regressions identify the best predictors of first 
semester GPA, retention (fall to spring) and persistence (fall to fall) respectively, from 
among the independent variables included in this study. 

Utilizing SPSS binary logistic regression the dependent variable is specified and 
then student demographic variables (gender, ethnicity and age group) and college 
variables (College, FTPT, Reading Score and Financial Aid Group) are entered into block 
one and the regression is run. Interactions among those variables reaching statistical 
significance and all other variables are computed. 

Then just the demographic variables are loaded into block one and the regression 
is run again after which only demographic variables of significance retained in block one. 
Next all interaction variables specific to the demographic variables are loaded in block 
two and the regression run. Following this, only variables in blocks one and two that 
reach statistical significance are retained. 

Following this, college variables are loaded in block three and the regression run. 
Only variables in blocks one, two and three that reach statistical significance are retained. 
Then all interaction variables specific to the college variables are loaded in block four 
and the regression run. Only variables in blocks one, two, three and four that reach 
statistical significance are retained. 

The regression is run one last time utilizing only the remaining significant 
variables and then collinearity diagnostics are run on the remaining variables with the 
expectation that all remaining variables achieve a tolerance of <.01 and a Variance 
Inflation Factor (VIF) of <10. 
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The fourth research questions concerns Fall Semester GPA, thus the dependent 
variable for the first regression analysis. For this dependent variable the exact process 
described above is followed. The sixth research question concerns retention, thus the 
dependent variable for a second regression analysis and a similar process is followed, 
only now GPA becomes an independent variable along with the other college variables. 
The eighth research question concerns persistence, thus the dependent variable in this last 
regression analysis. The same procedure is followed as for retention only now retention 
also becomes an independent variable along with GPA and the other college variables. 


The design of these regression analysis are modeled after the Input (I) - 
Environment (E) - Output (O) model of assessment first developed my Alexander W. 
Astin (1991) and later modified by Dr, Michael E. Moore, Dr. William E. Knight and 
myself while we were at Georgia Southern University (1994). 


Student 

Background 

Characteristics 

► 

Student 

Experiences 

► 

Student 

Outcomes 


X it 



Institutional 

Resources 

► 

Institutional 

Processes 

> 

Institutional 

Outcomes 


External Environment 
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Positive Outcomes 


Results 


Among the 5,144 new students in the fall of 2006, 222 (4.3%) were awarded 
financial aid in the form of a Pell Grant only; 599 (1 1.6%) had aid other than a Pell 
Grant; 1,316 had a Pell Grant plus other aid; and 3,007 (58.5%) received no financial aid. 
As shown in Table 1, students who received financial aid in the form of a Pell Grant and 
no other aid have a significantly lower number of positive outcomes than any of the other 
groups (X 2 = 29.851, df= 1 5 , pc. 05). Positive outcomes occurred for 53 students or 
23.9% of the of the Pell Grant only recipients while the success rates for other financial 
aid groups range from 32.1% to 36.7%. The overall success rate for the total cohort is 
32.6%. 

Table 1 Positive Outcomes 



Pell Grant only 

Aid other 
than Pell 
Grant 

Pell Grant 
and other aid 

No financial 
aid 

Total Cohort 


N 

% 

N 

% 

N 

% 

N 

% 

N 

% 

Initial Cohort 

222 

4.3% 


11.6% 

IBB 

25.6% 


58.5% 

5144 













Certificate w/o Transfer 

0 

0.00% 

ED 

1.67% 

mm 

0.99% 

26 

0.86% 

49 

0.95% 

Associates Degree w/o Transfer 

5 

2.25% 


6.51% 

mn 

5.40% 

141 

4.69% 

256 

4.98% 

Certificate and Transferred 

0 

0.00% 

■ 

0.00% 

\ 2 

0.15% 

5 

0.17% 

7 

0.14% 

Associates Degree and Transferred 

4 

1.80% 

m 

4.17% 

mn 

3.12% 

MM 

4.12% 

194 

3.77% 

Transferred w/o Award 

24 

10.81% 


14.69% 


11.25% 

437 

14.53% 

697 

13.55% 

Still Enrolled w/o Award >= 30 hours 

20 

9.01% 


9.68% 


11.17% 

247 

8.21% 

472 

9.18% 












|Total Success 

53 

| 23.87% | 

E8B1 

36.73% 

BE 

32.07% 


32.59% 

1675 

32.56% 


Markers of Academic Progress 


Progression through College-level Mathematics 

Among students for whom their first math course was Pre-Algebra, successful 
completion rates for College-level Mathematics are low irrespective of financial aid 
group, ranging from 20.4% to 27.0%. There are no significant differences among the 
various groups (F = .860, df = 3, p>.05). Table 2 provides the frequencies and 
percentages for students in each financial aid category. 
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Table 2 Started with Pre-Algebra 


Started Pre-Algebra 

Pell Grant 
only 

Aid other 
than Pell 
Grant 

Pell Grant 
and other aid 

No financial 
aid 

Total Cohort 


m 

% 

□ 

% 

N 

% 

N 

% 

N 

% 

Initial Cohort 


4.3% 


11.6% 

1316 

25.6% 


58.5% 

5144 













Started Pre-Algebra 

IBEl 

48.6% 


40.7% 

631 

47.9% 

891 

29.6% 

1874 

36.4% 

Success Pre-Algebra 

m 

25.7% 

SJ* 

29.7% 

415 

31.5% 


20.2% 

1256 

24.4% 

Enrolled Elementary Algebra 


24.3% 


25.7% 

347 

26.4% 

475 

15.8% 

IBE 

20.0% 

Success Elementary Algebra 


15.8% 


16.9% 

221 

16.8% 

316 

10.5% 

673 

13.1% 

Enrolled College-Level Math 


15.8% 


15.4% 

214 

16.3% 

282 

9.4% 

623 

12.1% 

Success College-Level Math 


9.9% 


11.0% 

156 

11.9% 

206 

6.9% 

450 

8.7% 












College-Level Math Success 


20.4% 


27.0% 


24.7% 


23.1% 


24.0% 


Among students for whom their first math course was Elementary Algebra, the 
successful completion rates for College-level Mathematics are better for all financial aid 
groups, ranging from 37.7% to 43.1% and there are no significant differences among the 
financial aid groups (F = 1.128, df = 3, p>.05). Table 3 provides the frequencies and 
percentages for students in this category. 

Table 3 Started with Elementan’ Algebra 


Started Elementary Algebra 

Pell Grant 
only 

Aid other 
than Pell 
Grant 

Pell Grant 
and other aid 

No financial 
aid 

Total Cohort 


m 

% 

o 

% 

N 

% 

N 

% 

N 

% 

Initial Cohort 


4.3% 


11.6% 

1316 

25.6% 


58.5% 

5144 













Started Elementary Algebra 


14.9% 


26.5% 

297 

22.6% 

847 

28.2% 

1336 

26.0% 

Success Elementary Algebra 


9.0% 


17.7% 

199 

15.1% 

584 

19.4% 


17.7% 

Enrolled College-Level Math 

m 

9.5% 


15.0% 

158 

12.0% 

458 

15.2% 

727 

14.1% 

Success College-Level Math 

KB 

6.3% 

m 

11.2% 

128 

9.7% 

319 

10.6% 

528 

10.3% 












College-Level Math Success 


42.42% 


42.14% 


43.10% 


37.66% 


39.52% 


Among students for whom their first math course was College-Level Math, the 
successful completion rate for College-Level Mathematics is even greater for all financial 
aid groups ranging from 62.5% to 66.1%, and there are no significant differences among 
the financial aid groups (F = .133 df = 3, p>.05). Table 4 provides the frequencies and 
percentages for students in this category. 

Table 4 Started with College-level Math 


Started College Level Math 

Pell Grant 
only 

Aid other 
than Pell 
Grant 

Pell Grant 
and other aid 

No financial 
aid 

Total Cohort 


m 

% 

o 

% 

N 

% 

N 

% 

N 

% 

Initial Cohort 


4.3% 

BEE 

11.6% 

1316 

25.6% 


58.5% 

5144 













Started College Level Math 


14.4% 

hh 

19.4% 

184 

14.0% 

543 

18.1% 

875 

17.0% 

Success College Level Math 


62.5% 

m 

64.7% 

118 

64.1% 

359 

66.1% 

572 

65.4% 

College Level Math Success 


9.01% 


12.52% 


8.97% 


11.94% 


11.12% 
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Among all students who successfully completed College-level Math, irrespective 
of what their first math course was, there are significant differences among the various 
financial aid groups (F=3.142, df = 3, p<.05). 


Table 5 Success in College-Level Math 


Financial Aid Group 

Mean 

Std. Deviation 

N 


Pell Grant Only 

0.25 

0.44 

222 

Other than Pell Grant 

0.35 

0.48 

599 

Pell Grant and Other 

0.31 

0.46 

1316 

No Financial Aid 

0.29 

0.46 

3007 

Total 

0.30 

0.46 

5144 

Source 

Type III Sum of Squares 

df 

Mean Square 

F 

Sig. 

Corrected Model 

1.983a 

3.0 

0.661 

3.142 

0.024 

Intercept 

197.8 

1.0 

197.826 

940.661 

0 

FinancialAidGroup 

2.0 

3.0 

0.661 

3.142 

0.024 

Error 

1081.0 

5140.0 

0.21 



Total 

1550.0 

5144.0 




Corrected Total 

1083.0 

5143.0 





a. R Squared = .002 (Adjusted R Squared = .001) 







95 % Confidence Interval 

(I) Financial Aid 
Group 

(J) Financial Aid 
Group 

Mean 

Difference 

(I-J) 

Std. Error 

Sig. 

Lower Bound 

Upper 

Bound 

Pell Grant Only 

Other than Pell Grant 

-.09* 

0.036 

0.008 

-0.17 

-0.02 


Pell Grant and Other 

-0.1 

0.033 

0.11 

-0.12 

0.01 


No Financial Aid 

0.0 

0.032 

0.191 

-0.1 

0.02 

Other than Pell Grant 

Pell Grant Only 

.09* 

0.036 

0.008 

0.02 

0.17 


Pell Grant and Other 

0.0 

0.023 

0.065 

0 

0.09 


No Financial Aid 

.05* 

0.021 

0.009 

0.01 

0.09 

Pell Grant and Other 

Pell Grant Only 

0.1 

0.033 

0.11 

-0.01 

0.12 


Other than Pell Grant 

0.0 

0.023 

0.065 

-0.09 

0 


No Financial Aid 

0.0 

0.015 

0.448 

-0.02 

0.04 

No Financial Aid 

Pell Grant Only 

0.0 

0.032 

0.191 

-0.02 

0.1 


Other than Pell Grant 

I 

o 

L/t 

* 

0.021 

0.009 

-0.09 

-0.01 


Pell Grant and Other 

0.0 

0.015 

0.448 

-0.04 

0.02 


Based on observed means. 

The error term is Mean Square(Error) = .210. 

*. The mean difference is significant at the .05 level. 


The mean success rate for recipients of Pell Grants (25%) is significantly lower 
than that of students receiving aid other than Pell Grants (35%). The mean success rate 
for recipients of aid other than Pell Grants (35%) is significantly higher than the mean for 
students receiving only Pell Grants (25%) and students receiving no financial aid (29%). 
The mean success rate for recipients of Pell Grants and other aid is 31% and not 
statistically different from the other three groups. The mean success rate for students 
receiving no financial aid is 29% and is significantly lower than recipients of aid other 
than Pell Grants (35%) 
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There is a statistically significant, correlation among the number of terms a 
student waits before taking his/her first math course and eventually attempting (r=.057, 
p<.05) and successfully completing (r=.065, p<.05) College-Level Math. The earlier the 
first math course is taken the better the chance of a student has of attempting and 
successfully completing College-Level Math irrespective of financial aid group. 

Table 6 Success in College-Level Math and First Dev Math Course 



Mean 

Std. Deviation 

N 


Fir stDevMathT erm 

1.52 

1.31 

3956 

Attempt College Level Math 

0.41 

0.491 

5144 

Success College Level Math 

0.3 

0.459 

5144 



FirstDevMathT erm 

Attempt College 
Level Math 

Success College 
Level Math 

FirstDe vMathTerm 

Pearson 

Correlation 

1 

.057** 

.065** 


Sig. (2-tailed) 


0 

0 


N 

3956 

3956 

3956 

Attempt College 
Level Math 

Pearson 

Correlation 

.057** 

1 

791** 


Sig. (2-tailed) 

0 


0 


N 

3956 

5144 

5144 

Success College 
Level Math 

Pearson 

Correlation 

.065** 

791** 

1 


Sig. (2-tailed) 

0 

0 



N 

3956 

5144 

5144 


**. Correlation is significant at the 0.01 level (2 -tailed). 


Among all students new in the fall of 2000 and followed over time for six years, 
1,059 students, or 20.6% of the population, did not attempt any mathematics course while 
enrolled in the Community College System. There were significant differences among the 
financial aid participants (X 2 = 1.82E2, df=9, pc. 05). The group most likely to attempt 
mathematics is students receiving no financial aid. From this group, 726 students (24.1% 
of students with no aid and 14.1% of the total cohort) attempted no math course. Table 5 
provides the frequencies and percentages for students in each financial aid category. 

Table 7 Attempted No Mathematics Course 



Pell Grant 
only 

Aid other 
than Pell 
Grant 

Pell Grant 
and other aid 

No financial 
aid 

Total Cohort 


Q 

% 

o 

% 

N 

% 

N 

% 

N 

% 

Initial Cohort 


4.3% 


11.6% 

1316 

25.6% 


58.5% 

5144 

4.3% 












Took No Math (within financial aid group) 


22.1% 


13.4% 

204 

15.5% 

726 

24.1% 


20.6% 

Took No Math (within cohort ) 


1.0% 


1.6% 

204 

4.0% 

726 

14.1% 


20.6% 
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First Semester GPA 


Overall 63.0% of all new students had a GPA of >=2.0 at the conclusion of the 
first semester. Overall recipients of Pell Grants are more likely to have a GPA <2.0 at the 
end of the first semester. Even after controlling for reading scores, the same dynamic 
holds (F=3.273, df = 3, p<.05). 

Table 8 First Semester GPA 




Pell Grant Only 

Other than Pell 
Grant 

Pell Grant and 
Other 

No Financial 
Aid 

Total 



N 

% 

N 

% 

N 

% 

N 

% 

N 

% 

GPA 

<2.0 

93 

44.3% 

195 

34.2% 

495 

39.2% 

1032 

36.0% 

1815 

37.0% 


>=2.0 

117 

55.7% 

376 

65.8% 

769 

60.8% 

1831 

64.0% 

3093 

63.0% 


Total 

210 


571 


1264 


2863 


4908 



Table 8 First Semester GPA (Continued) 


Source 

Type III Sum of Squares 

df 

Mean Square 

F 

Sig. 

Corrected Model 

18.814a 

4 

4.704 

20.499 

0 

Intercept 

176.327 

1 

176.327 

768.479 

0 

Reading Group 

16.385 

1 

16.385 

71.411 

0 

Financial Aid Group 

2.253 

3 

0.751 

3.273 

0.02 

Error 

1124.991 

4903 

0.229 



Total 

3093 

4908 




Corrected Total 

1143.805 

4907 





a. R Squared = .016 (Adjusted R Squared = .016) 


Although there is an apparent relationship between financial aid status and First 
Semester GPA, financial aid status on its own is not a significant predictor once other 
demographic and college variables are accounted for. Hispanic Males, students at Capital 
Community College in Financial Aid Group 3 (Pell Grant and Other Aid), Hispanic 
students at Capital Community College, and Black students at Naugatuck Valley 
Community College, are less likely than students not in these categories to achieve a 
GPA of 2.0 or higher. 

Students at Housatonic in Reading Group 5 (No Reading Score on File), students 
at Northwestern Connecticut Community College in Reading Group 2 (2 nd Quartile), full- 
time students at Norwalk Community College, along with female and Asian students at 
Naugatuck Valley Community College, are more likely than students not in these 
categories to achieve a GPA of 2.0 or higher. 
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Table 9 Predictors of First Semester GPA 


Predictor 

B 

SE 

Wald 

Exp(B) 

Sig 

Tolerance 

VIF 

NVBIk 

-0.951 

0.293 

10.539 

0.386 

** 

0.960 

1.042 

CAFA3 

-0.869 

0.205 

18.067 

0.419 

** 

0.984 

1.016 

CAHsp 

-0.669 

0.305 

4.825 

0.512 

** 

0.981 

1.019 

MHsp 

-0.455 

0.126 

13.101 

0.635 

** 

0.970 

1.031 

HORG5 

0.298 

0.140 

4.555 

1.347 

** 

0.983 

1.017 

NKFT 

0.573 

0.158 

13.207 

1.774 

** 

0.992 

1.008 

NVF 

0.589 

0.136 

18.680 

1.803 

** 

0.942 

1.061 

NWRG2 

1.622 

0.748 

4.697 

5.062 

** 

0.998 

1.002 

NVAsian 

1.635 

0.743 

4.839 

5.132 

** 

0.990 

1.010 

Constant 

0.518 

0.034 

236.330 

1.679 

** 




p<.05; ** p< .01 
Retention (Fall to Spring) 


Overall 69.0% of all new students were retained from fall to spring of their first 
academic year. Overall students receiving aid other than Pell Grants (75.3%) and those 
receiving Pell Grants and other aid (75.9%) are retained at a higher rate than other 
groups. Students receiving no financial aid are the least likely to be retained. The same 
dynamic holds even after controlling for reading scores (F=18.918, df = 3, p<.05). 


Table 10 Retention 


■ 

Pell Grant Only 

Other than Pell 
Grant 

Pell Grant and 
Other 

No Financial Aid 

Total 


N 

% 

N 

% 

N 

% 

N 

% 

N 

% 

No 

69 

31.1% 

148 

24.7% 

317 

24.1% 

106? 

35.4% 

1597 

31.0% 

Yes 

153 

68.9% 

451 

75.3% 

999 

75.9% 

1944 

64.6% 

3547 

69.0% 

Total 

222 


599 


1316 


3007 


5144 


Source 

Type III Sum of Squares 

df 

Mean 

F 

Sig. 

Corrected Model 

24.517 

4 

6.129 

29.255 

0.000 

Intercept 

488.941 

1 

488.941 

2333.719 

0.000 

ReadingGroup 

10.167 

1 

10.167 

48.529 

0.000 

Financial AidGroup 

11.891 

3 

3.964 

18.918 

0.000 

Error 

1076.680 

5139 

0.210 



Total 

3547.000 

5144 




Corrected Total 

1101.197 

5143 





a. R Squared = .022 (Adjusted R Squared = .022) 


Although there is an apparent relationship between financial aid status and 
retention, financial aid status on its own is not a significant predictor of retention once 
other demographic and college variables are accounted for. At Gateway Community 
College students in Financial Aid Group 3 (Pell Grant and Other Aid) along with Black 
and Hispanic students, are less likely to be retained than students not in these categories. 
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Students in reading Group 1 (1 st and lowest Quartile), part-time students with a 
first semester GPA of 2.0 or higher, full-time students in Reading Group2 (2 nd Quartile), 
students at Capital Community College in Financial Aid Group 3 (Pell Grant and Other 
Aid), and students in Financial Aid Group 2 (Aid other than Pell) and Financial Aid 
Group 3 with a first semester GPA of 2.0 or higher, are more likely to be retained than 
students not in these categories. 

Table 11 Predictors of Retention 


Predictor 

B 

SE 

Wald 

Exp(B) 

Sig 

Tolerance 

VIF 

GWFA3 

-0.694 

0.208 

11.090 

0.500 

** 

0.620 

1.612 

GWBIk 

-0.558 

0.201 

7.701 

0.572 

** 

0.752 

1.330 

GWHsp 

-0.501 

0.248 

4.103 

0.606 


0.837 

1.195 

ReadGroupl 

0.292 

0.081 

12.936 

1.338 

** 

0.954 

1.048 

GPAHPT 

0.413 

0.089 

21.676 

1.512 

** 

0.945 

1.059 

R2FT 

0.667 

0.117 

32.252 

1.949 

** 

0.938 

1.066 

CAFA3 

0.854 

0.260 

10.790 

2.348 

** 

0.985 

1.015 

GPAHFA2 

1.839 

0.186 

97.757 

6.293 

** 

0.982 

1.018 

GPAHFA3 

2.269 

0.159 

204.661 

9.666 

** 

0.894 

1.119 

Constant 

0.338 

0.043 

62.603 

1.402 

** 




p<.05; ** p< .01 
Persistence (Fall to Fall) 


Overall 49.2% of all new students persisted from the fall of their first academic 
year to the fall of the second academic year. Overall students receiving Pell Grants only 
persisted at a lower rate (39.2%) than other groups. The same dynamic holds even after 
controlling for reading scores (F=4.793, df = 3, pc. 05). 

Table 12 Persistence 




Pell Grant Only 

Other than Pell 
Grant 

Pell Grant and 
Other 

No Financial Aid 

Total 



N 

% 

N 

% 

N 

% 

N 

% 



Persistence 

No 

135 

60.8% 

288 

48.1% 

627 

47.6% 

1561 

51.9% 

2611 

50.8% 


Yes 

87 

39.2% 

311 

51.9% 

689 

52.4% 

1446 

48.1% 

2533 

49.2% 


Total 

222 


599 


1316 


3007 


5144 


Source 

Type III Sum oi 

f Squares 

di 

1 

Mean Square 

] 

F 

Sig. 

Corrected Model 

10.368 

4 

2.592 

10.444 

0.000 

Intercept 

228.732 

1 

228.732 

921.682 

0.000 

ReadingGroup 

6.018 

1 

6.018 

24.250 

0.000 

FinancialAidGroup 

3.568 

3 

1.189 

4.793 

0.002 

Error 

1275.337 

5139 

0.248 



Total 

2533.000 

5144 




Corrected Total 

1285.704 

5143 





a. R Squared = .008 (Adjusted R Squared = .007) 


Although there is an apparent relationship between financial aid status and 
persistence (fall to fall), financial aid status on its own is not a significant predictor of 
retention once other demographic and college variables are accounted for. Students in 
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Reading Group 4 (4 th Quartile or Highest) are less likely to persist than students not in 
Reading Group 4. Students in Reading Group 2 (2 nd Quartile) and students with a GPA 
of 2.0 or higher are more likely to persist than students not in these categories. 

Table 13 Predictors of Persistence 


Predictor 

B 

SE 

Wald 

Exp(B) 

Sig 

Tolerance 

VIF 

ReadGroup4 

-0.080 

0.085 

0.902 

0.923 


0.904 

1.107 

ReadGroup2 

0.198 

0.093 

4.521 

1.218 


0.918 

1.089 

GPAHigh 

2.019 

0.069 

864.832 

7.534 

** 

0.171 

5.842 

Constant 

-0.259 

0.050 

26.624 

0.772 

** 

0.174 

5.764 


p<.05; ** p< .01 


Summary 

Although some differences in performance exist among the various financial aid 
participants and non-participants, these differences cannot be attributed to the financial 
aid groups alone. Variables, both demographic and college specific, are interacting with 
each other to form significant combinations that allow for estimates of the likelihood of 
attaining known academic markers such as successful completion of college-level math, 
attaining a first semester GPA >= 2.0, being retained fall to spring and persisting fall to 
fall. The question now becomes why (or why not) students in various categories 
identified by logistic regression, are succeeding in the first semester, being retained and 
persisting, regardless of financial aid participation? For example, why are students in the 
highest reading group less likely to persist? Are they transferring early? In general, why 
are Hispanic Males less likely to achieve a first semester GPA of >= 2.0 than other 
students? Why are Black and Hispanic students especially at three of the urban colleges 
performing so differently than other students? What other factors in addition to those 
revealed in this study, are impacting students and how? What strategies can be taken to 
further enhance the success of students? 
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Table 14 Summary of Predictor Variables 


GPA >= 2.0 

NVcc Black 

less likely 

CAcc Pell Grant and Other Aid 

less likely 

CAcc Hispanic 

less likely 

Male Hispanic 

less likely 

HOcc No Reading Score on Record 

more likely 

NKcc Full-Time 

more likely 

NVcc Female 

more likely 

NWcc Reading Score in 2nd Quartile 

more likely 

NVcc Asian 

more likely 


Retention 

GWcc Pell Grant and Other Aid 

less likely 

GWcc Black 

less likely 

GWcc Hispanic 

less likely 

Lowest Reading Group 

more likely 

GPA >=2.0 Part-time 

more likely 

Reading Score in 2nd Quartile Full-time 

more likely 

CAcc Pell Grant and Other Aid 

more likely 

GPA >=2.0 Aid Other than Pell 

more likely 

GPA >=2.0 Pell Grant and Other Aid 

more likely 


Persistence 

Highest Reading Group 

less likely 

Reading Score in 2nd Quartile 

more likely 

GPA >=2.0 

more likely 
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ENHANCING PARENTS' ROLE IN HIGHER EDUCATION ASSESSMENT 


Anne Marie Delaney 
Director of Institutional Research 
Babson College 

Introduction. This paper presents a model designed to enhance parents’ role in higher 
education assessment. The model is based on a study that compared parents' and students' 
evaluation of undergraduate education. Major research questions include the following: 

To what extent do parents think the undergraduate program enhanced their child's 
personal, intellectual and career relevant abilities? 

How satisfied are parents with the academic, social, career preparation and campus 
life aspects of their child's college experience? 

What specific aspects of parents' evaluation predict their overall satisfaction with the 
undergraduate program? 

How do parents' evaluations compare with their children's in terms of the College's 
impact on students' abilities and satisfaction with aspects of college life? 

Literature Review 

Wartman and Savage (2008) offer a compelling rationale for involving parents in the life 
and work of higher education institutions. 

“By bringing parents into the educational equation on personal, social and economic 
issues, administrators gain a partner who has the most at stake in their student’s well- 
being. Moreover, parents can add to institutional messages by infusing them with family 
or personal history. Increasingly, schools consider parents as not only an audience but 
also a stakeholder in the messages they deliver to students.” (p. 90) 

Developments in society and in parent-child relationships have enhanced parents’ role in 
higher education. Kirwan (2007) observes that our nation has developed an educational 
deficit in relation to the rest of the industrialized world and we must recapture the sense of 
higher education as a common good worthy of greater investment of public funds. Further, 
since paying for a college education is one of the most expensive purchases parents incur, 
they have every right to expect access to information about costs and accountability for 
outcomes, including graduation rates, job-placement and graduate school acceptance 
statistics. Scott and Daniel (2001) note that parents matter to higher education because of the 
increasing costs of a college education; parents' expectations that they be involved; and 
changing family dynamics. 

A cultural shift in parent-child relationships has resulted in increasing involvement of 
parents in the lives of today’s college students. Jackson and Murphy (2005) cite the 
evolution of parent-child relationships into partnerships as a major factor accounting for a 
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change in the role of parents in higher education. Wartman and Savage (2008) identify five 
factors that may contribute to the increasing involvement of parents: generational 
differences; cost of college; use of technology; changes in parenting style and demographics. 
Results from the 2007 National Survey of Student Engagement offer empirical evidence of 
parents’ greater involvement in students’ lives. Seven out of ten students reported they 
communicated very often with at least one parent or guardian during the academic year. 
Students talked with their mothers about personal issues, academic performance, and family 
matters, and they communicated most often with their fathers about academic matters. 

Previous research studies have investigated parents’ role in various aspects of higher 
education, including college choice, goals of undergraduate education and, to a limited 
extent, evaluation of college outcomes. Some studies have compared parents’ and students’ 
perspectives. 

With regard to college choice. Smith and Bers (1989) found that parents were involved in 
information gathering but were not necessarily decision-makers. In terms of college choice 
criteria, Warwich and Mansfield (2003) determined that both parents and students identified 
academics as the most important criterion and shared four of the same top criteria - 
academics, tuition, financial aid and friendly atmosphere. In contrast, Brokemier and 
Seshadri (1999) discovered significant differences between students’ and parents’ college 
choice criteria. Students attributed significantly more importance to social life, friends 
attending an institution, and athletic programs, while parents assigned more importance to 
cost, program of study, facility quality, academic reputation, and campus safety. Both 
students and parents most frequently identified parents as the major influence in the college 
choice decision. 

Turrentine at al. (2000) explored parents’ goals for their students’ overall college 
experience. Results revealed that parents’ top goals were: job preparation, quality education, 
maturity/independence, graduation, fun/enjoyment, academic success and 
friendships/networks. 

A select number of studies have compared parents' and students' perspectives on quality, 
goals and outcomes. Litten and Hall (1989) found that parents were more likely to select 
faculty related indicators, while students were more likely to select program related factors in 
assessing institutional quality. With regard to goals of undergraduate education, Papish 
(2000) discovered that parents attributed more importance to a mix of academic and student 
development goals - including managing psychological stress and developing skills for self- 
sufficient living, while students placed more value on gaining a sense of interdependence 
with others and getting involved in community service. 

Bisset et al. (1999) compared the views of students, parents and faculty members 
regarding the importance of educational goals for undergraduate education. Comparative 
findings for parents and students revealed that parents placed more importance on academic 
goals related to science and technology, humanities, social and behavioral sciences, 
intellectual abilities and personal and social development. However, both parents and 
students highly valued career related goals, including acquiring skills for a special job and 
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getting a good job after graduation. Parents and students also shared an appreciation for the 
personal and social development that a college education affords; they ranked gaining a sense 
of independence and gaining self-confidence among their top five goals for undergraduate 
education. 

Browne, Kaldenberg and Browne (1998) compared parents’ and students’ evaluation of 
an undergraduate business education program. Results revealed significant positive 
correlations between parents' and students' ratings on faculty teaching skills and advising. 
Parent and student ratings were also positively correlated on overall satisfaction, willingness 
to recommend the college and satisfaction with the value of the education. 

Methodology 

Data Source. The population for this study included 281 parents of seniors who 
graduated in May 2007. A survey was administered primarily on the Internet during the 
2008 spring semester. Responses were received from 155 parents, yielding a response rate of 
55 percent. The paper presents results for all respondents and for a subset of 122 parents 
whose responses were matched with their children's responses. 

Methods of Analysis. Correlation analyses examined relationships between parents' 
evaluation of specific aspects of the college experience and their overall satisfaction and 
willingness to recommend the College. Regression analyses identified specific aspects of 
parents' evaluation that predicted their overall satisfaction and willingness to recommend the 
College. Analyses were conducted with individual items and scale scores. 

Development of Scales. Common factor analysis or the principal axis factor method was 
employed to identify the dimensions underlying parent responses to the survey. Items 
included in the scales had factor loadings of .4 or higher. Three scales were created 
representing: parents' assessment of the undergraduate program's impact on students' 
abilities; parents' satisfaction with the academic program and parents' satisfaction with 
campus social life. Items in the Assessment of Impact on Abilities scale include parents’ 
perception of the program’s impact on their child’s ability to: understand moral and ethical 
issues; relate well to people of different races; think analytically and logically; write 
effectively; communicate well orally and exercise leadership. The Satisfaction with the 
Academic Program scale reflects parents’ satisfaction with faculty attitude and academic 
advising. The Satisfaction with Campus Life scale represents parents’ satisfaction with extra- 
curricular programs and campus social life on. 


Table 1 presents the statistical properties of the scales including the mean and 
standard deviation for the total group and the alpha reliability coefficients, which range from 
.73 to .82, indicating moderate to high reliability. 
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Table 1. Statistical Properties of the Scales 


Scale Names 

Mean 

S.D. 

Reliability 

No. of 
Items 

Range of 
Responses 
Low-High 

Assessment of Impact on Abilities 

3.57 

.44 

.82 

6 

1 -4 

Satisfaction with Academic Program 

3.32 

.62 

.80 

2 

1 -4 

Satisfaction with Campus Life 

3.24 

.64 

.73 

2 

1 -4 


Results 

Parents’ Assessment of the Undergraduate Program 

Perceived Impact on Abilities. Figure 1 graphically displays the percent of parents who 
perceived that the College greatly enhanced their child's abilities in each area. 

Figure 1. Percent Rating Students' Abilities 'Greatly' Impacted 


0% 20% 40% 60% 80% 


Acquire New Skills and Knowledge 
Think Analytically and Logically 
Communicate Well Orally 
Gain In-Depth Knowledge of a Field 
Leadership Ability 



Use Computers 
Understand Moral/Ethical Issues 
Relate Well to People of Diff. Races 
Write Effectively 



100% 

I 

90% 


As shown, 90 percent of the parents reported that the College greatly impacted the ability 
to acquire new skills and knowledge and 72 to 76 percent perceived that the College greatly 
enhanced the ability to think analytically, communicate well orally, gain in-depth knowledge 
of a field and exercise leadership. In contrast, only 49 percent thought that the College 
greatly enhanced the ability to write effectively. 
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Satisfaction with College Experiences. Responses reflect considerable variance in parent 
satisfaction with three areas of their child’s college life: the academic program, student 
services and campus life. With respect to the academic program, 76 percent were very 
satisfied with the quality of business courses, while only 32 percent were very satisfied with 
academic advising. Regarding student services, 44 percent were very satisfied with the 
financial aid package, while only 26 percent were very satisfied with career services. In the 
area of campus life, 58 percent were very satisfied with campus safety, but only 26 percent 
were very satisfied with student housing. 

Overall Assessment. Two questions addressed parent’s overall assessment of their child's 
undergraduate education. One question elicited level of satisfaction and another question 
focused on parents’ willingness to recommend the College to a high school senior. Results 
revealed that 69 percent were very satisfied and 26 percent were satisfied with their child’s 
education. Some 76 percent reported they definitely would and 17 percent probably would 
recommend the College to a high school senior. 

Table 2 identifies significant correlates of parents’ overall satisfaction. As shown, 
parents who perceived a greater impact on their child's ability to acquire new knowledge and 
skills and leadership expressed higher overall satisfaction. Similarly, those who were 


Table 2. Significant Correlates of Parents' Overall Satisfaction 


Correlates 

Correlation 

Impact on Abilities 

Acquire New Skills and Knowledge 

42 *** 

Exercise Leadership 

33 *** 

Satisfaction with College Experiences 

Student Housing 

.35 *** 

Faculty Attitude 

34 *** 

Satisfaction with College Offices 

Career Services Office 

51 *** 

Counseling Services Office 

45 *** 

Health Service Office 

41 *** 

Housing Office 

39 *** 

Class Dean's Office 

33 *** 

Satisfaction with the College's Communication with Parents 

Overall Communication 

39 *** 

E-mail Communication 

36 *** 


*** p < .001 

more satisfied with student housing, faculty attitude, career services, other student services 
offices and the College’s communication with parents also expressed higher satisfaction with 
their child's education. 
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Table 3 identifies significant correlates of parents' willingness to recommend the College. 
As shown, parents who perceived that the College enhanced their child's ability to acquire 
new skills and knowledge; understand moral and ethical issues; exercise leadership; think 
analytically and logically; and gain in-depth knowledge of a field were more willing to 
recommend the College. Also, those who were more satisfied with student housing, extra- 
curricular programs, faculty attitude and the College's communication were more willing to 
recommend the College to a high school student. 


Table 3. Correlates of Parents' Willingness to Recommend the College 


Correlates 

Correlation 

Impact on Abilities 

Acquire New Skills and Knowledge 

.56 *** 

Understand Moral and Ethical Issues 

41 *** 

Exercise Leadership 

3g *** 

Think Analytically and Logically 

37 *** 

Gain In-depth Knowledge of a Field 

.36 *** 

Satisfaction with College Experiences 

Student Housing 

4g *** 

Extra-Curricular Programs 

42 *** 

Faculty Attitude 

40 *** 

Satisfaction with the College's Communication with Parents 

Overall Communication 

39 *** 

E-mail Communication 

.36 *** 


*** p < .001 


Predicting Parents ’ Overall Satisfaction. Regression analysis addressed the question, 
“What specific aspects of parents' evaluation predict their overall satisfaction with the 
undergraduate program?” Results, presented in Table 4, identify parents' perception of the 
College's impact on their child's ability to acquire new skills and knowledge; satisfaction 
with academic advising; and satisfaction with career services as significant predictors of 
overall satisfaction. The R 2 with these variables in the model is .51, indicating that these 
variables explain 5 1 percent of the variance in parents’ overall satisfaction with their child’s 
undergraduate education. 
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Table 4. Predicting Parents' Overall Satisfaction 


Predictors 

Beta 

Coefficient 

t 

Ratio R2 F Ratio 

Perceived impact on ability to acquire new 



skills and knowledge 

.43 

5.33*** 

Satisfaction with academic advising 

.31 

3 39 *** 

Satisfaction with career services 

.24 

2.74** 


.51 28.06*** 


** p < .01;*** p < .001 

Predicting Parents ’ Willingness to Recommend the College. Regression results, 
displayed in Table 5, also identified perceived impact on the ability to acquire new skills and 
knowledge as a significant predictor of willingness to recommend the college. In addition, 
satisfaction with student housing and extra-curricular programs were identified as significant 
predictors. The R" of .49 indicates that these variables explain 49 percent of the variance in 
parents' willingness to recommend the College. 


Table 5. Predicting Parents' Willingness to Recommend the College 


Predictors 

Beta 

Coefficient 

t 

Ratio R2 F Ratio 

Perceived impact on ability to acquire new 



skills and knowledge 

.46 

6.85*** 

Satisfaction with student housing 

.28 

3.97** 

Satisfaction with extra curricular programs 

.19 

2.62** 


.49 40.18*** 


**p< .01;*** p< .001 

Comparison of Parents' and Students' Assessment 

Matched pair t tests were conducted with responses from 122 parents and their children to 
answer the question, “How do parents’ evaluations compare with their children's in terms of 
the College's impact on students' abilities and satisfaction with aspects of college life?” 

Perceived Impact on Abilities. Table 6 presents results regarding perceived impact on abilities. 

As shown, parents' ratings are consistently higher than those of their children. The largest 
significant differences involve the ability to relate well to people of different races and to 
understand moral and ethical issues. Parents' mean ratings indicate they perceive a moderate to 
great impact, while their children perceive only a little to moderate impact. The smallest differences 
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involve leadership ability and the ability to acquire new skills and knowledge; both parents and 
seniors perceive a moderate to great impact on these abilities. 


Table 6. Significant Differences between Parents' and Seniors' 
Assessment of Babson College's Impact on Students 'Abilities 


Mean Ratings 


Abilities! 

Parents 

Seniors 

Difference 

T test 

Relate Well to People of Different Races 

3.39 

2.48 

.91 

10.01 *** 

Understand Moral and Ethical Issues 

3.48 

2.65 

.83 

9.20 *** 

Think Analytically and Logically 

3.77 

3.24 

.53 

6.55 *** 

Write Effectively 

3.37 

2.91 

.46 

5.39 *** 

Use Computers 

3.58 

3.26 

.32 

3.65 *** 

Gain In-depth Knowledge of a Field 

3.69 

3.41 

.28 

3.56 *** 

Communicate Well Orally 

3.72 

3.49 

.23 

3.45 *** 

Acquire New Skills and Knowledge 

3.91 

3.71 

.20 

3.55 *** 

Leadership Ability 

3.67 

3.50 

.17 

2.35 * 


f These ratings are based on the scale: 1 'Not at AH' 2 'A Little' 3 'Moderately' 4 'Greatly'. 
* p < .05; *** p < .001 


Satisfaction with College Experiences. Table 7 presents parents' and seniors' mean 
satisfaction ratings regarding various aspects of the college experience. As shown, parents' 
ratings are generally higher than those of their children. The largest difference involves 
satisfaction with social life on campus. The parents’ mean rating of 3.14 indicates satisfied, 
while the senior mean rating of 2.28 indicates dissatisfied. Compared with seniors, parents 
also report substantially higher satisfaction ratings on student financial services and student 
housing; the parent mean ratings suggest satisfied, while the senior mean ratings range between 
dissatisfied and satisfied. There are no significant differences 

between parents' and seniors' mean satisfaction ratings on faculty attitude, quality of business 
courses and campus safety. Both parents and their children report mean ratings that range 
between satisfied and very satisfied on these college experiences. 
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Table 7. Significant Differences between Parents' and Seniors' 
Satisfaction with Students' College Experiences 


Mean Ratings 


College Experiencesf 

Parents 

Seniors 

Difference 

T test 

Social Life on Campus 

3.14 

2.28 

.86 

9.06 *** 

Student Financial Services 

3.13 

2.62 

.51 

4.45 *** 

Student Housing 

3.13 

2.66 

.47 

5.56 *** 

Academic Advising 

3.17 

2.85 

.32 

4.22 *** 

Financial Aid Package 

3.03 

2.73 

.30 

2.56 * 

Extra-curricular Programs 

3.31 

3.11 

.30 

2.08 * 

Career Services 

3.09 

2.80 

.29 

2.83 ** 

Faculty Attitude 

3.42 

3.43 

-.01 

n.s. 

Quality of Business Courses 

3.74 

3.78 

-.04 

n.s. 

Campus Safety 

3.57 

3.42 

.15 

n.s. 


t These ratings are based on the scale: 1 'Very Dissatisfied' 2 'Dissatisfied' 3 'Satisfied' 4 'Very Satisfied'. 


* p < .05; ** p < .01; *** p < .001 

Overall Evaluation. Table 8 presents the results of t tests comparing parents' and seniors' 
overall assessment of the undergraduate program. On overall satisfaction, the parents' mean 
of 3.64 is significantly higher than the seniors' rating. The second set of mean ratings is 
based on slightly different, but related questions. Parents were asked, "Would you 
recommend the College to a high school senior interested in a career in business?" Seniors 
were asked, "If you had a chance to relive your college experience, would you choose to 
attend the College again?" The parents' mean rating of 4.62 is also significantly higher than 
the seniors' mean rating. 


Table 8. Significant Differences between Parents' and Seniors' 
Overall Assessment of the College's Undergraduate Program 



Mean Ratings 



Parents Seniors 

Difference T test 





Overall Satisfaction with the Education 

3.64 

3.28 

.36 

4.83 

Would Recommend or Choose the College Again 

4.62 

3.70 

.92 

8.89 


*** p < .001 

Note : Overall satisfaction mean ratings range from 1 ‘very dissatisfied’ to 4 ‘very satisfied’. 
Response options for recommending or choosing the same college range from 
1 ‘definitely not’ to 5 ‘definitely would’. 
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The following selected comments offer insight into the reasons for parents’ positive 
evaluation of their child’s undergraduate education, including: excellent education, enriching 
student life experiences, responsive student services and effective communication with parents. 


From day one until the day my daughter graduated from the College, it has been the best 
investment 1 have made in my life and in my daughter’s life. 

My son got an outstanding education and grew more then I could have imagined in four short 
years. He was thrilled with the academics. 

His participation on the swimming and diving team for four years, his athletic records and 
achievements, coupled with his active involvement in the admissions office for three years 
helped make him a strong candidate for summer internships which eventually led to his 
existing job in the consulting arena. 

While my son attended the College, we had a serious family medical emergency and found 
the College’s response, locating my son and informing him of the need to call home, to be 
speedy and very caring. Thereafter, professors and staff expressed concern and care, which 
helped him weather the crisis. 

I'm impressed with the College's efforts at communication and support. Being so far away 
and my son being determined to be as self-reliant as possible, actual interaction with the 
faculty and administration has been limited. 

Recommendations 

Based on the research findings, the following types of policy recommendations were 
designed to promote the College’s strengths and address areas for improvement. 

• In communicating with prospective students' parents, share current parents' 
confirmation of their choice reflected in their willingness to recommend the College. 

• Utilize parent testimony to portray the undergraduate program as a wise investment. 

• Promote parents' positive evaluation of the program's impact on students' abilities. 

• Publicize parent's high level of satisfaction with the undergraduate program. 

• Intensify the program's focus on enhancing students' ability to write effectively. 

• Review and enhance the undergraduate academic advising program. 

• Closely monitor student housing to ensure high guality maintenance. 

• Ensure that all College offices are responsive to parents' inguiries and concerns. 

Discussion 
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This study represents an effort to involve parents in assessing higher education’s 
effectiveness. The research is based on the belief that parents ought to be involved given the 
investment they make; their expectation to be involved; and the significant role they play in 
their children’s lives (Kirwan, 2007; Scott & Daniel, 2001). 

Results revealed that the vast majority of parents rated the program highly for enhancing 
students’ ability to think analytically; gain in-depth knowledge of a field; and lead 
effectively. In contrast, only a minority considered that the program greatly enhanced 
students’ ability to write effectively. Parents expressed varying levels of satisfaction with 
aspects of the college experience. While the majority were very satisfied with the quality of 
business courses and campus safety, only a minority were very satisfied with academic 
advising and student housing. In terms of overall assessment, close to 70 percent or more 
were very satisfied with their child’s education and would definitely recommend the College 
to a high school senior. 

Comparative analyses revealed that parents and students rated the program highly for 
enhancing students’ ability to acquire new skills and knowledge and develop leadership 
ability. However, parents reported substantially and significantly higher ratings for the 
program’s impact on students’ ability to relate well to people of different races and to 
understand moral and ethical issues; parents perceived a moderate to great impact, while their 
children perceived only a little to moderate impact. Parents also reported significantly higher 
satisfaction with campus social life; the parent mean rating indicated satisfied, while the 
student mean rating indicated dissatisfied. 

There were no significant differences between parents and students in terms of satisfaction 
with faculty attitude and quality of business courses; both parents and students reported 
positive mean ratings between satisfied and very satisfied. Browne, Kaldenberg and Browne 
(1998) also found significant positive correlations between parents’ and students’ ratings on 
faculty teaching skills and satisfaction with the overall value of the education in an 
undergraduate business program. 

Parents and students reported positive mean ratings on the two outcome measures 
reflecting an overall assessment of the undergraduate program. On overall satisfaction, the 
parent mean rating was close to very satisfied, while the student mean rating was close to 
satisfied. In terms of recommending or choosing the same college again, parents reported a 
mean close to definitely would, while students reported a mean close to probably would. 

Regression analyses identified significant predictors of parents’ overall satisfaction and 
willingness to recommend the College. Perceived impact on students’ ability to acquire new 
skills and knowledge was a significant predictor of both outcomes. Other predictors of these 
outcomes varied. Satisfaction with academic advising and career services significantly 
predicted overall satisfaction, while satisfaction with student housing and extracurricular 
programs significantly predicted willingness to recommend the College. 

Previous research with students also identified a different set of predictors for overall 
satisfaction and the likelihood of choosing the same college again. Delaney (2005) found 
that student satisfaction with the quality of business courses; satisfaction with faculty 
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attitude; and the perception of enhanced ability to think analytically significantly predicted 
overall satisfaction. In contrast, satisfaction with campus life and satisfaction with campus 
safety significantly predicted the likelihood of choosing the same college again. Thus, 
research with parents and students identifies factors related to academic preparation as 
significant predictors of overall satisfaction with undergraduate education, while factors 
related to student life are significant predictors of willingness to recommend or choose the 
same college again. 
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Abstract 

Institutional researchers often understand “survey design” as a process 
in which researchers translate broad research questions into specific 
survey items. Here, we go beyond this conception and search for ways 
to improve the experience of taking a web-based engagement survey. 

We present evidence from an experiment that this process can result in 
increased response rates and better quality data. 

Introduction 

Cornell University has administered a survey of student engagement four times since 
2003. This survey, titled the Enrolled Student Survey or the ESS, was developed in 
consortium with 30 other elite institutions and included over 250 individual items. Over the 
course of the four survey administrations, Cornell’s response rate declined substantially, 
from 47% in 2003 to 29% in 2007. 

In an effort to address declining response rates on our campus, we considered the role 
of survey instrument design and how it can contribute positively or negatively to the 
experience of completing a web-based survey of student engagement. In this paper, we 
describe how we: reduced the overall length of the survey; enhanced its visual appeal; and 
increased students’ perceptions that the survey was relevant to their experiences. Evidence 
from an experiment on our campus indicates that the new survey generates a higher response 
rate with better quality data. 

Conceptual Approach 

Existing research on survey design emphasizes that potential respondents evaluate 
signals from the survey instrument itself in deciding whether to respond. We focused on the 
following four considerations: 
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1) Survey length. The perceived burden of responding to a survey is tied directly to its 
overall length (Bogen, 1996; Handwerk, Carson, & Blackwell, 2000) as well as the number 
of questions that appear per web screen (Schonlau, Fricker, & Elliott, 2002). In the course of 
this project, we reduced a survey with over 250 items to one with fewer than 100. In 
addition, we searched for ways to signal that the survey was not burdensome to complete, 
and that progress through the survey was swift (e.g. Schonlau et al., 2002). 

2) Survey content. Researchers are advised to use a “top down” approach to survey 
construction where underlying research questions are identified and relevant survey questions 
follow. While there is considerable merit in this theory driven approach, we note that 
surveys enjoy higher response rates when their contents are seen as relevant to the 
respondents’ own experiences and values (Dillman, 1991; Groves et al., 2000; Groves, 
Presser & Dipko, 2004). To enhance the salience of our survey, we asked our interview 
participants what they thought of the questions we had developed and what they wanted to 
tell us in a survey. Consequently, we added a handful of new questions to the instrument. 

3) Visual appeal. When survey design is viewed as relating narrowly to the choice and 
wording of questions, insufficient thought may be given to the process of converting those 
items into a web-based survey instrument. Yet extant research suggests the visual design of a 
web-based survey affects response rates (Couper, Traugott & Lamias, 2001; Dillman, 2000; 
Dillman, Tortora, Conradt & Bowker, 1998). In this project, we paid a great deal of attention 
to the overall look and feel of the survey, and responded to student feedback regarding the 
appearance of the pages. 

4) Delivery of survey results. Providing respondents with survey results can help 
build rapport with respondents (Cook, Heath & Thompson, 2000; Groves, Singer & Coming, 
2000; Ribisl et al., 1996). In this project, we took advantage of what the web makes possible, 
using Cold Fusion software to incorporate instant results into the survey-taking process. In 
this way, we immediately rewarded survey respondents with a handful of findings at key 
points. 


Process 

In consortium with 30 other elite institutions, Cornell University had administered the 
ESS four times since 2003. That survey included over 250 individual items, and had gone 
through one substantial revision (before the 2007 administration). While we had achieved a 
respectable 47% response rate in 2003, our 29% response rate in 2007 enhanced our desire to 
lessen the burden the survey placed on students. 

We held a day-and-a-half face-to-face meeting with like-minded institutional research 
colleagues from peer institutions to establish the foundations for moving the project forward. 
Specifically, we held a sustained dialogue concerning, first, the core objectives of our 
institutions and their intersections and, second, the criteria we should use in deciding whether 
or not a specific item should be included in this survey. 
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As an illustrative example of “core objectives”: there was consensus that our 
institutions conscientiously emphasize vibrant intellectual communities (beyond the 
classroom), including: residential communities that foster the exchange of ideas, interactions 
with diverse kinds of students, and meaningful discussions with faculty. Thus, a survey that 
would enable us to hold ourselves accountable for the things we believe we should be doing 
would include measures relevant to intellectual communities. 

The criteria we adopted to make decisions as to whether or not any specific item would 
appear on the new survey included: 

■ The survey item should clearly relate to our shared research agenda. 

■ The survey item reflects the principles of quality survey design. 

■ The survey is of interest to several institutions. 

■ Previous use of the survey item produced results of practical importance on 
our campuses. 

■ Prior analyses have demonstrated that the item has a proven track record 
statistically: it has sufficient variance; it does not suffer from high item- 
nonresponse; it is not redundant with another indicator; it is an important 
predictor of key outcomes; and/or outcomes have changed over time such that 
timely data provide new insights. 

■ The wording and interpretation of new survey items have been thoroughly 
explored and pre-tested with students. 

■ There is utility in time-series data: we should not change ESS question 
wording or response categories without a compelling reason. That being said, 
the utility of legacy items needs to be assessed. 

With careful attention to both our core objectives and our agreed upon criteria for 
inclusion, an engaged working group reduced the 250 item survey to a core of approximately 
75 measures (including 13 measures — like race and gender — which could be supplied from 
administrative files when that is a possibility). 

With input from a graphic designer, we worked with a Cornell -based survey service to 
mock-up a version of the survey instrument to use for the purposes of testing and evaluation. 
We then recruited students — with a $20 payment for a one-hour interview — to come to our 
office to take the survey and provide their candidate feedback about the experience of taking 
the survey. In total, we interviewed nineteen undergraduate students. After the feedback 
received in the first ten interviews, we made substantial revisions to both the aesthetics and 
the substance of the evolving survey instrument. This revised instrument was then pretested 
with nine more students, with only minor changes resulting. 

As a final step in our research, we took the survey instrument that resulted from this 
process — now dubbed “PULSE” — and tested it head-to-head with the original ESS 
instrument. In this experiment, we drew a random sample of 300 summer session students 
and randomly assigned 100 to take the original ESS just as it had been administered in the 
spring semester of 2007 and 200 students to take the new PULSE. Results from that 
experiment are described in the section titled, “Did it Work?,” below. 
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Survey Redesign: A Look at Page 1 

Figures 1 and 2 are screen captures of the ESS (the original instrument) and PULSE 
(the revised instrument) respectively. In glancing across the figures, perhaps the first thing to 
note is that the first page of the ESS is notably longer than the first page of the PULSE. This 
is partly a reflection of the fact that the entire ESS is longer than the PULSE, but it is also the 
case that the single longest page of the ESS is the first page, with a total of 42 individual 
items. In contrast, we conscientiously elected to make the first page of the PULSE 
particularly short in an attempt to signal with the first impression that the PULSE was not a 
particularly lengthy or burdensome endeavor. 

A second difference between the instruments is that we replaced the simple mechanical 
“progress bar” on the ESS with a more elaborate header on the PULSE. The PULSE header 
is not navigable; clicking on it will not take you to a different page of the survey. Rather, the 
goal of the header is to alleviate anxiety about “where the survey is going” by providing a 
succinct outline of the entire scope of the instrument. 

Third, on the basis of student feedback, we replaced the black grid used on the ESS 
with a white grid and increased the spacing between items. 

Fourth, and perhaps most substantively, the two instruments use different lead-off 
questions. This is not because the surveys differed substantively in content; in fact, the 
PULSE includes several of the items used to kick-off the ESS. Rather, this was a strategic 
choice in recognition of work by Dillman and others which indicates that the first question on 
a survey signals the nature of the survey and its salience to the respondent. With the PULSE, 
we elected to begin by asking for students’ generalized opinions of their experience at the 
university rather than simply asking for unreflective reporting of behaviors. Further, given 
substantial evidence that academic engagement is positively correlated with survey response 
rates, we sought to avoid reinforcing this tendency by introducing the survey with a section 
on “Academics.” 

What Student Didn’t Want to Tell Us, and What They Did 

In our initial extended interviews with students taking the PULSE, the single most 
common complaint with the survey had to do with the ha nk of “time budget” questions. On 
the ESS, this bank is launched with the following stem: “During the current school year, 
approximately how many hours do you spend during a typical week doing the following 
activities.” As illustrated in Figure 3, this stem is followed by 28 individual items, and 
students are asked to respond regarding the time spent on each item using one of ten response 
categories. 

In the PULSE, we limited ourselves to asking about eight behaviors, but students still 
told us unequivocally that it was both daunting to be faced with a “sea of bubbles” ten — or 
even eight — columns wide and quite burdensome to try to have any precision in their 
estimates of how much time they spent “exercising” or “hanging out.” Indeed, those of us 
who have had the experience of having to account for our time use — such as for reporting 
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purposes at work — may be able to relate to their irritation on this note. Accordingly, on the 
PULSE we sacrificed some precision and limited ourselves to just five broad response 
categories (see Figure 4). We were able to get more coverage out of five categories by 
dividing the time-use questions across two different banks, with behaviors tending to occupy 
a large amount of time (such as studying) using a different response scale than behaviors 
typically consuming less time (such as community service). 

As a part of the sustained interviews with our nineteen pre-testers, we asked 
respondents if there were things that they felt we omitted from the survey — things that they 
would want to tell us about their undergraduate experience that we had not asked about. 

More than half of our pre-testers indicated in one way or another that they wanted to 
tell us more about what is like to be a Cornell student. Specifically, they wanted to tell us 
that they work hard, that they sacrifice sleep, and that they are stressed. Further, they wanted 
to tell us not just about binge drinking — a single question on the survey — but about the role 
of alcohol in social life and how they have managed to navigate that. Finally, they wanted to 
tell us about the importance of Greek life on Cornell’s campus. In response to this feedback, 
we dismissed our own hesitancy to inquire into quite personal issues such as drinking and 
mental health and added new questions that tapped these aspects of undergraduate life. 

Web Surveys: More than a Paper Survey Put on the Web 

As we searched for ways to improve the experience of taking the new PULSE survey, 
we invited ourselves to think about how we could use the fact that is a web-based survey to 
our advantage. As one of our early collaborators suggested, we should think about a web 
survey as being “more than just a paper survey put on the web.” We seized three 
opportunities. 

First, entirely generic survey instruments that ask about “your institution” or “your 
college” may reflect the economies of scale inherent in the mass production of paper surveys. 
With the web, however, the costs of customization are substantially less. In the PULSE, we 
ask about satisfaction with “Cornell University” rather than with “your institution,” and we 
expect that our peer institutions will do similarly. 

Second, we have tended to use random draws for prizes as incentives for our surveys. 
Our past practice had been to wait until the survey closes to draw winners. The trouble with 
this practice, however, is that it separates the behavior and its reward (stimulus and response) 
by days or even weeks. Indeed, we have contacted prize winners in the past who have told us 
they had completely forgotten they had taken the survey! While this might be the best 
possible model for drawing winners with a paper survey, the web makes it possible to 
program lottery selection right into the survey itself. Thus, for the PULSE, we indicated that 
every tenth respondent would know instantly if they had won $20. By notifying respondents 
at the time they complete the survey , we create the possibility that they will share their 
delight over winning with other students thereby encouraging others to participate in the 
survey as well. 
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Third, there is considerable evidence in the literature that respondents want to hear 
about the results of the study in which they participated. Indeed, our students told us as 
much as well. However, the effective delivery of study findings has proven to be more 
problematic than one might immediately assume. Further, as with post-survey lottery 
drawings, the lag between taking the paper survey and receiving a summary of findings can 
be substantial. Here again, the power of computing can be used to address the issue. At 
three points in the PULSE, we offered students “instant results” about a question they had 
already answered. For example, after responding to a question about their own satisfaction 
on one page of the survey, respondents are told on the next page, “X% of students who have 
taken the PULSE so far also rated their entire educational experience at Cornell as 
‘Excellent.” And, upon hitting “submit” on the last page of the survey, respondents are 
provided a full page of bar charts and pie graphs portraying the responses “thus far” to key 
items of interest on the PULSE. 

We also found that on our campus, students were sensitive to the notion of going to a 
URL outside of Cornell to take a Cornell survey. By hosting the survey ourselves - rather 
than using a single survey provider for the whole consortium - we were able to allay 
concerns that the survey was anything other than a Comeli-based effort to inform Cornell 
administrators about the Cornell experience. 

Did it Work?: Impact on Response Rates and Data Quality 

As described above, in order to test whether our efforts would make a difference in the 
response rate to the survey, we drew a random sample of 300 summer session students and 
randomly assigned 100 to take the original ESS and 200 students to take the new PULSE. 
Nearly identical emails were sent from the Dean of Students inviting the students to take the 
surveys; the only differences in the text were the names of the surveys and the estimated time 
it would take to complete the survey. Thus, 100 students were asked to complete the ESS 
with instructions that the “The questionnaire typically takes 20 to 25 minutes to complete,” 
and 200 students were invited to take the PULSE with the comment “The survey typically 
takes around 10 minutes to complete.” 

Under any definition of “survey response,” the PULSE fared better than the ESS. For 
example, if we define a student as a respondent if he or she answered just one or more 
questions, the PULSE attained a response rate of 47%, while the ESS response rate was just 
39%. 


More remarkable, however, is the data we observed on persistence through the survey, 
as is illustrated in Figures 5 and 6. Aside from the sentence in the invitation email cluing 
students to the estimated length of time to take the survey, there is little reason to expect that 
fewer students would even open the ESS as compared to the PULSE. Thus we were 
somewhat surprised to find any difference at all in the percent of students who even entered 
the survey (48% of those taking the revised PULSE as compared to 42% taking the original 
ESS). 5 The difference grew from there, however: 45% of the sample asked to take the 


5 The programming for the ESS did not allow us to observe how often a student followed the link to the survey 
and then left without any further action. However, we did observe that three of the 110 students opened the 
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PULSE reached the final page of the survey, compared to just 36% of those taking the ESS. 
Moreover, among those who responded, nearly all (94%) of those answering the PULSE 
provided answers for 90% or more of the questions asked, compared to just 60% of ESS 
respondents doing so. 

An analysis not shown here suggests that PULSE respondents were also less select vis- 
a-vis non-respondents than the ESS respondents as compared to non-respondents. That is: 
the gap between the GPAs and SAT scores of respondents and non-respondents was 
substantially smaller for the PULSE group than for the ESS sample. This suggests that the 
PULSE data may be more representative of the population as a whole. 

Implications 

Inarguably, the content of a survey should be dictated, first and foremost, by the 
research questions the instrumented is intended to address. Clearly articulating the core 
research objectives underpinning a survey and establishing criteria to evaluate the utility of 
specific survey items are two important steps in developing surveys that are both relevant to 
our purposes and concise. Like other researchers (e.g., Bogen, 1996; Handwerk et al., 2000; 
Schonlau et al., 2002), we found that students are quite sensitive to the effort required to 
participate in a survey - whether measured by estimated time for completion, total number of 
questions, or the complexity of the response categories employed in particular questions. 
Dropping survey response rates suggest that our students are less and less willing to give up 
their time to complete a survey. Together, these points argue for being very sure that we are 
asking the right questions on our surveys and only those questions that really need to be 
asked. 

But survey design should not be driven by content considerations alone. We may know 
what we want to ask our students, but how we present those questions within a web 
instrument may well affect the likelihood that students will choose to participate in the 
survey and the quality of the data we ultimately collect. Consistent with past research (e.g., 
Couper et al., 2001; Dillman, 2000), our study found that visual design elements - such as 
the borders and spacing used in questions, and the number of questions presented per page - 
elicited strong reactions from our pretesters. And ultimately, differences in survey length 
and visual design seem to have accounted for significant differences in the proportion of 
students responding to the original and revised versions of our survey, as well as differences 
in the proportion of survey questions completed. Visual design elements are most under our 
control when we are developing local survey instruments. Even when using externally- 
authored web-based instruments where the content has been predetermined, there may be 
opportunities for visually presenting this content in ways that can positively affect survey 
participation and for hosting the survey on a local domain. If so, the cost of programming 
such changes would seem to be well worth the potential benefits in survey response and 
completion rates. 


survey, went to the bottom of page one without answering any questions and clicked the submit button there 
without going on to answer any later questions or submit any other pages. Only one student acted analogously 


with the PULSE. 
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Figure 1. Page 1 of Enrolled Student Survey (ESS) 



Cornell University 


Progress: | 2 3 4 5 6 7 8 9 Complete 


Academics 


1 . Purina the current school year , how often have you done each of the following? 


Course or classroom experiences 

Never 

Occasionally 

Often 

Very Often 

Worked on a class assignment, project, or 
presentation with other students 

r 

r 

r 

r 

Worked on a paper or project that required integrating 
ideas or information from various sources 

c 

r 

r 

r 

Participated in class discussions 

r 

r 

r 

r 

Made a formal presentation in class 

r 

r 

r 

r 

Prepared for class with an informal study group 

r 

r 

r 

r 

Prepared a major written report, such as a thesis, 
honors project, or significant research paper 

r 

r 

r 

c 

Taken a course with a component in a large lecture hall 

r 

r 

r 

r 

Had a graduate student as a teaching assistant 

r 

r 

r 

r 

Discussed intellectual ideas with other students 
outside of class 

r 

r 

r 

r 

Research, Writing and the Arts 

Never 

Occasionally 

Often 

Very Often 

Conducted research using historical archives, surveys, 
field work, or other primary sources on a project 

r 

r 

r 

r 

Prepared two or more drafts of a paper or assignment 
before turning it in 

r 

r 

r 

r 

Published or presented a paper or research off campus 

r 

r 

r 

r 

Attended a concert or other music event, on or off 
campus 

r 

r 

r 

r 

Participated in a music activity (orchestra, chorus, etc.) 
on or off campus 

r 

r 

r 

r 

Participated in art (pottery, painting, etc.) or theatrical 
production (acted, danced, etc.) on or off campus 

r 

r 

r 

r 

Written a poem, story, or other creative writing for a 
class 

r 

r 

r 

r 

Read or discussed the opinions of art, music, or drama 
critics 

r 

c 

r 

r 

Math, Science & Technology 

Never 

Occasionally 

Often 

Very Often 

Completed an experiment or project using the scientific 
method 

r 

r 

r 

r 

Practiced to improve your skill using a piece of 
laboratory equipment 

r 

r 

r 

r 

Read articles about scientific or mathematical concepts 
not assigned for a class or class project 

r 

r 

r 

r 

Used mathematical terms or formulae to express a set 
of relationships 

r 

r 

c 

r 

Used a computer to produce visual displays of 
information (charts, graphs, spreadsheets, etc.) 

r 

r 

r 

r 

Used a computer to analyze data (statistics, 
forecasting, etc.) 

r 

r 

r 

r 

Developed a Web page 

r 

r 

r 

r 

Faculty 

Never 

Occasionally 

Often 

Very Often 

Worked harder than you thought you could to meet 
the instructor's standards or expectations 

r 

r 

r 

r 

Worked with a faculty member on a research project 
for credit 

O 

r 

r 

r 

Worked with a faculty member on a research project 
not for credit 

r 

r 

r 

r 

Discussed your career plans and ambitions with a 
faculty member 

r 

r 

r 

r 

Discussed your academic work with a faculty member 

r 

r 

c 

r 

Discussed your course selection plans with a faculty 
member 

r 

r 

r 

r 

Had intellectual discussions with a faculty member 
outside of class 

r 

r 

r 

r 

Interacted with a faculty member at a social event 

c 

r 

r 

r 

Library 

Never 

Occasionally 

Often 

Very Often 

Asked a librarian or staff member for help in finding 
information on some topic 

r 

r 

r 

r 

Worked on a project that used a special collection of 
books, materials, or papers 

r 

r 

r 

r 

Found something interesting while browsing in the 
library 

r 

r 

r 

r 

Used the library as a quiet place to read or study 
materials you brought with you 

r 

r 

r 

r 

Used a computer to retrieve materials from a library or 
source not at this institution 

r 

r 

r 

r 


2. Durino the current school year , how much has your coursework emphasized the following mental 
activities? 



Very 

little 

Some 

Quite 
a bit 

Very 

much 

Memorizing facts, ideas or methods from your courses 
and readings so you can repeat them in pretty much 
the same form. 

r 

r 

r 

r 

Analyzing the basic elements of an idea, experience, 
or theory, such as examining a particular case or 
situation in depth and considering its components. 

r 

r 

r 

r 

Synthesizing and organizing ideas, information, or 
experiences into new, more complex interpretations 
and relationships. 

r 

r 

r 

r 

Making judgments about the value of information, 
arguments, or methods, such as examining how others 
gathered and interpreted data and assessing the 
soundness of their conclusions, 

r 

r 

r 

r 

Applying theories or concepts to practical problems or 
in new situations. 

r 

r 

r 

r 


Next » | 

Click Next to save your entries on this page and proceed to the next page. 
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Figure 2. Page 1 of the PULSE 


The Cornell PULSE 


Perceptions of Undergraduate Life and Student Experiences 


Your 
satisfaction 
with Cornell 


Your 

academic 

involvement 


Your 
interactions 
with faculty 


Changes 
in your skills 
and abilities 


Extracurricular 
and social 
involvement 


Your 
health and 
well-being 


Your 

background 


How would you evaluate your entire educational experience at Cornell University? 

Poor 

r 


Excellent 

r 


Good 

r 


Fair 

r 


How satisfied are you with the following aspects of your Cornell experience so far? 



Very 

Generally 

Generally 

Very 


dissatisfied 

dissatisfied 

satisfied 

satisfied 

Overall quality of instruction 

r 

r 

r 

r 

Academic advising 

r 

r 

r 

r 

Administration's responsiveness to student concerns 

r 

r 

r 

r 

Social life on campus 

r 

r 

r 

r 

Sense of community on campus 

r 

r 

r 

r 

Sense of community where you live during the academic year 

r 

r 

r 

r 


Submit (page 1 of 7) » 


See how quickly this goes?? 
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Figure 3. The Time-Budget Bank on the ESS 


7. Purina the current school year , approximately how many hours do you spend during a typical week doing 
the following activities? 


Academic 

None 

Less 
than 2 
hours 

2-4 

hours 

5-10 

hours 

11-15 

hours 

16-20 

hours 

21-25 

hours 

26-30 

hours 

31-40 

hours 

More 

than 

40 

hours 

Attending scheduled classes or labs 

r 

r 

r 

r 

r 

r 

r 

r 

r 

r 

Working on scheduled courses outside of class or lab 
(i.e., homework) 

r 

r 

r 

r 

r 

r 

r 

r 

r 

r 

Doing other academic work (e,g. thesis, internship, 
research) 

r 

r 

r 

r 

r 

r 

r 

r 

r 

r 

Health and Athletics 

None 

Less 
than 2 
hours 

2-4 

hours 

5-10 

hours 

11-15 

hours 

16-20 

hours 

21-25 

hours 

26-30 

hours 

31-40 

hours 

More 

than 

40 

hours 

Participating in Intercollegiate Athletics (during practice 
and playing season) 

r 

r 

r 

r 

r 

r 

r 

r 

r 

r 

Participating as a cheerleader, mascot, or member of 
the marching band 

r 

r 

r 

r 

r 

r 

r 

r 

r 

r 

Playing on intramural athletic or club sport team(s) 

r 

r 

r 

r 

r 

r 

r 

r 

r 

r 

Exercising or using a fitness or weight room 

r 

r 

r 

r 

r 

r 

r 

r 

r 

r 

Competing informally on a team or sport (e,g., 
"pick-up" game) 

r 

r 

r 

r 

r 

r 

r 

r 

r 

r 

Eating meals 

r 

r 

r 

r 

r 

r 

r 

r 

r 

r 

Extracurricular Activities 

None 

Less 
than 2 
hours 

2-4 

hours 

5-10 

hours 

11-15 

hours 

16-20 

hours 

21-25 

hours 

26-30 

hours 

31-40 

hours 

More 

than 

40 

hours 

Participating in student government 

r 

r 

r 

r 

r 

r 

r 

r 

r 

r 

Working on a campus newspaper 

r 

r 

r 

r 

r 

r 

r 

r 

r 

r | 

Working on a literary magazine 

r 

r 

r 

r 

r 

r 

r 

r 

r 

r 

Participating in a Fraternity or Sorority ("Eating Club" or 
"Secret Society") 

r 

r 

r 

r 

r 

r 

r 

r 

r 

r 

Playing or singing in an orchestra, band, choral or 
other musical group 

r 

r 

r 

c 

r 

r 

r 

r 

r 

r 

Working on a theatrical production 

r 

r 

r 

r 

r 

r 

r 

r 

r 

r 

Participating in a religious organization or religious 
service 

r 

r 

r 

r 

r 

r 

r 

r 

r 

r 

Volunteering in the community 

r 

r 

r 

r 

c 

r 

r 

r 

c 

c 

Participating in a political organization 

r 

r 

r 

r 

c 

r 

r 

r 

r 

c 

Participating in a minority or ethnic organization 

r 

r 

r 

r 

c 

r 

r 

r 

r 

c 

Participating in another organized student activity or 
club 

c 

r 

r 

r 

c 

r 

r 

r 

r 

c 

Job 

None 

Less 
than 2 
hours 

2-4 

hours 

5-10 

hours 

11-15 

hours 

16-20 

hours 

21-25 

hours 

26-30 

hours 

31-40 

hours 

More 

than 

40 

hours 

Working at a job that was part of a financial aid 
package (e.g., work -study program) 

r 

r 

r 

r 

r 

r 

r 

r 

r 

r 

Working at a job that was not part of a financial aid 
package 

r 

r 

r 

r 

r 

r 

r 

r 

r 

r 

Recreation 

None 

Less 
than 2 
hours 

2-4 

hours 

5-10 

hours 

11-15 

hours 

16-20 

hours 

21-25 

hours 

26-30 

hours 

31-40 

hours 

More 

than 

40 

hours 

Socializing and talking with friends 

r 

r 

r 

r 

r 

r 

r 

r 

r 

r 

Watching TV 

r 

r 

r 

r 

r 

r 

r 

r 

r 

r | 

Using a computer for non-academic activity (e.g., video 
games, Facebook, IM) 

r 

c 

r 

r 

r 

r 

r 

r 

r 

r 

Reading for pleasure 

r 

r 

r 

r 

r 

r 

r 

r 

r 

r | 

Visiting with a family or friend off campus 

r 

r 

r 

r 

r 

r 

r 

r 

r 

r 

Relaxing by yourself 

r 

r 

r 

r 

r 

r 

r 

r 

r 

r 
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Figure 4. The Time-Budget Banks on the PULSE 


Page 2: 


Purina the typical week this academic year, approximately how many hours do you spend on your 
academic work? 


Hours per week: 



0-10 

11-15 

16-20 

21-30 

30+ 

[s2qla] Attending scheduled classes or labs 

c 

r 

r 

r 

r 

[s2qlb] Working on academic work outside of class or lab (e.g., 
homework, research project, etc.) 

r 

r 

r 

r 

r 


Page 5: 


Purina the typical week this academic year, approximately how many hours do you spend doing the 
following activities? 

Hours per week 



None 

1-5 

6-10 

11-20 

21+ 

[s5qla] Working for pay 

r 

r 

r 

r 

r 

[s5qlb] Physical fitness (exercise, sports, etc.) 

r 

r 

r 

r 

r 

[s5qlc] Community service 

r 

r 

r 

r 

r 

[s5qld] Participating in other extracurricular 
activities (student organizations, fraternity or 
sorority, etc.) 

r 

r 

r 

r 

r 

[s5qle] Partying 

r 

r 

r 

r 

r 

[s5q If] Other relaxing or socializing (e.g., watching 

r 

r 

r 

r 

r 


TV, playing video games, hanging out) 
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Figure 6. Histogram of Proportion of Questions Answered, by Survey 
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MOVING DATA INTO ACTION: THE APPLICATION OF INSTITUTIONAL 
RESEARCH TO ENHANCE STUDENT SUCCESS 


Kimberly Puhala, M.S., Research Fellow 

and Kevin B. Murphy, Ph.D., Associate Director 

The Office of Institutional Research and Policy Studies 
The University of Massachusetts Boston 


Introduction 

The purpose of this research project was to examine factors related to student retention for 
students who entered the University of Massachusetts Boston (UMB) through the Directions 
for Student Potential (DSP) program. The DSP program is a six-week pre-matriculation 
summer program that provides intensive college preparation in reading, writing, 
mathematics, ESL, and study skills to students who don’t meet the admission requirements of 
the University, but show other academic potential. At the end of the successful completion 
of the DSP summer program, students are guaranteed admission to the University. The 
majority of DSP participants are low-income, first generation, minority students, who, 
according to the literature, tend to have lower retention and graduation rates than their peers. 
Researchers in UMB’s Office of Institutional Research and Policy Studies (OIRP) collected 
both qualitative and quantitative data on issues related to student retention, analyzed this 
data, and participated in the development of evidence-based plans for the implementation of 
an intervention designed to increase student success for this population. This paper will 
focus on the leadership role of the OIRP by delineating the research planning and the 
iterative data-based process that was utilized to develop the intervention. Implications for 
other IR professionals will also be discussed. 

Background 

College retention, persistence and degree attainment have been studied exhaustively over 
the past three or four decades. No single answer to questions of poor retention, persistence, 
and graduation rates has arisen. Perhaps the best-known retention theory is Tinto’s student 
integration model (1975, 1982, 1988, 1993, 2005) that suggests that retention, persistence, 
and graduation rates will improve if the student withdraws from the old life of family and 
neighborhood and becomes fully integrated into the society of the higher education 
institution. Indeed, much of the work in the area of postsecondary education retention either 
builds on or is set in contrast to Tinto’s theory. 

Most models attempt to examine traits that are intrinsic to the student, such as 
race/ethnicity, gender, and age. Most also include traits that may have been developed in 
earlier education such as standardized test scores and high school grade point average. A 
number of studies simply report differing success rates by race/ethnicity rather than using 
multivariate analyses to examine differences in success rates by race/ethnicity while holding 
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other factors equal. Tinto (1993) reported six-year completion/persistence rates (based on the 
1980 High School and Beyond study) for members of these groups who had entered four- 
year institutions immediately after high school as 60.7% for White non-Hispanics, 39.6% for 
Black non-Hispanics and 46.6% for Hispanics. Among the completers and persisters, White 
non-Hispanics were almost twice as likely to have graduated as were members of the other 
two groups. More recently, a report based on data from the Cooperative Institutional 
Research Program (CIRP) reported national six-year completion rates of 69.4% for Asian 
Americans, 52.2% for African Americans, 53.2% for Mexican- Americans, 44.3% for Puerto 
Ricans, and 61.6% for White non-Hispanics (Astin and Oseguera, 2002). 

The main college entrance standardized test scores used for college admissions are the 
ACT and SAT exams. While many studies use them as independent variables in their models 
of retention and persistence, the scores are interpreted to mean different things in different 
studies. Tinto (1993) and Astin & Oseguera (2002, 2005) use combined SAT scores as an 
indicator of institutional selectivity. Other studies use SAT or ACT scores as an indicator of 
student ability (Aitken, 1982; Astin & Oseguera, 2005; DesJardins et al, 2002; Pascarella, 
Terenzini, & Wolfle, 1986). Other studies have used the scores as an indicator of acquired 
skills (Aitken, 1982; Elkins et al, 2000). Still others use them as an indicator of general 
academic preparation (Murtaugh et al, 1999; Nettles et al., 1986). The target population of 
this study, students who entered the Directions for Student Potential (DSP) Program, partially 
qualified for the program on the basis of SAT scores that were below the range normally 
accepted at the university but were combined with other qualities that led the university to 
believe that the students could be successful. 

DSP students showed results that were not consistent with the results found in the 
literature for students with similar characteristics. For DSP students, the relationship of 
race/ethnicity and standardized test scores to retention was the opposite of what is usually 
found in the literature. These students were retained at higher rates than the non-DSP 
students who generally had significantly higher SAT scores. The DSP students are 
disproportionately minority, but over a number of cohorts often had higher retention rates 
than white non-Hispanic students who did not attend DSP. The contrast between outcomes 
predicted by the literature and the actual outcomes for the DSP students seemed important 
enough to explore further to ascertain whether something in the DSP program that was 
important to success could be expanded to a larger population. The planning year grant 
provided support for a year while a Community of Practice (COP) researched the program, 
tried to identify what made it successful, and developed a comprehensive plan to provide 
similar services to a larger population. 
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Charting the Course: The Roadmap 


Representatives from the Office of Institutional Research and Policy Studies (OIRP) at 
UMB played a critical leadership role in the COP in a number of ways. From the very 
beginning of the process, OIRP had a place at the table. The Associate Director of OIRP 
analyzed initial data that revealed that the DSP program participants have better retention 
rates than their non-DSP counterparts. In addition, the COP hired a part-time, grant-funded, 
project- specific research fellow, with a direct reporting line to OIRP rather than to the 
principal investigators. This allowed OIRP to lead the data collection and analysis process, 
such that the research was conducted within an IR context, rather than having IR answer data 
requests from the COP in a unidirectional process. This organizational structure also allowed 
for a seamless incorporation of additional IR data as the iterative process within the COP 
generated new questions. In addition, because the research fellow was 100% project-funded, 
there were no conflicting responsibilities, and the study was able to progress very quickly. 
Within a 6 month timeline, IRB approval was obtained, surveys were developed and 
administered, 6 focus groups were conducted, all quantitative and qualitative data were 
analyzed, and three study reports were generated. 

OIRP began playing a leadership role during the research planning stage. Survey 
instruments were developed and administered to first and third year DSP students. Rather 
than developing new survey instruments from scratch, OIRP developed surveys that were 
compatible with data fields that are already collected with routine student surveys, such as 
NSSE and CIRP. The outcome of this process was to create a longitudinal dataset with 
baseline information for the 2007 cohort of students, which will be usable after this specific 
project ends, through data merges with subsequent routine surveys that are conducted by 
OIRP. In addition, this strategy allowed for comparisons to be made between this specific 
cohort and other university cohorts who had completed the existing surveys. Also, existing 
data were merged with the new data that was collected. Data sources included pre-existing 
official University records of admissions and semester-by-semester enrollment data. In 
addition, a number of focus groups were planned that explored the experiences of the 2007 
cohort of DSP students with attention to students who were identified as particularly 
successful and others who were viewed as at-risk. Overall, the OIRP planning created a 
context of data triangulation that allowed for a comprehensive and holistic approach to 
understanding students’ experience as it relates to retention, and which allowed for merging 
with data from past and future planned student surveys. 

Re-routing with New Information 

The COP met monthly to discuss progress and to review the study findings. Through 
discussions at the COP meetings, we learned that there is a Student Support Services (SSS) 
program that is offered to many DSP students. The SSS program offers academic advising, 
loaner books and laptops, some financial support, a Foundations course which teaches time 
management and study skills, and unlimited tutoring to program participants. The SSS 
program is offered to first-generation, low income students, and there is space for 500 
students per year to qualify for the program, of which 100 are new students each year. The 
benefits of the SSS program are provided until students have accumulated 60 credits, at 
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which point they are no longer eligible for SSS services. Since the DSP population overlaps 
with the eligibility requirements for the SSS program, OIRP representatives questioned the 
amount of overlap between the DSP program and the SSS program. OIRP found that almost 
all of the SSS students enter through the DSP program, but only about half of the DSP 
students are offered SSS services. OIRP, concerned that the high overlap of these two 
programs may be impacting study results, gathered more information about the program by 
meeting with the SSS Program Director, and subsequently added SSS coding to the datasets. 
As a result, it was discovered that most of the effect that we were seeing in retention in DSP 
students was from the SSS component, not DSP alone. An example using the 2006 cohort 
(n=92 DSP Only and n=91 DSP & SSS) is shown in Figures 1. and 2. below. 

When the DSP group was not coded for SSS, retention rates were better for the DSP 
group than the Non-DSP group (82.0% vs. 70.7%, pc. 01), indicating that the DSP program 
enhances retention. However, once the SSS coding was incorporated, there were no 
differences between the DSP Only group and the Non-DSP group in retention rates (69.6% 
vs. 70.7%, p=.809), but there was a significant difference between the DSP & SSS group and 
the Non-DSP group (94.5% vs. 70.7%, pc.01). These results were presented to the COP, and 
a decision was made that the evidence pointed to focusing on the SSS component. 

Figure 1. One-Year Retention Rates: DSP Students vs. Non-DSP Students 


One- Year Retention 
DSP vs. Non-DSP 
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Figure 2. One-Year Retention Rates with Coding for Student Support Services (SSS) 


One-Year Retention w/ SSS Coding 



SSS 


Survey results also indicated that there was a need to further explore the impact of the 
SSS program. Approximately 60% of those DSP students who returned surveys were also in 
the SSS program, and of those in the SSS program, 66.1% rated the program as very 
important to their success and ability to continue in school. In addition, the Student Support 
Services program tied for first place with Computer Labs in rankings of student satisfaction 
with the services. Results are presented in Table 1. below. 


Table 1. Utilization of All Student Services Ranked by Importance for Success and Ability 
to Continue in School (n=l 10) 


Program/Service 

% 

Used 

n 

Used 

% Very 
Important 

% Very 
Satisfied 

Financial Support 

87.3% 

96 

72.9% 

38.6% 

Computer Fabs 

91.8% 

101 

67.3% 

52.5% 

Student Support Services 

58.2% 

64 

66.1% 

52.5% 

Fibrary Resources 

89.1% 

98 

61.2% 

41.8% 

Academic Support (tutors, workshops) 

79.1% 

87 

52.9% 

26.4% 

Advising/Counseling 

88.2% 

97 

51.5% 

37.1% 

Being a part of DSP 

96.4% 

106 

43.4% 

47.2% 

Student Fife Services 

68.2% 

75 

36.0% 

18.7% 

Club Affiliations or Student Organizations 

59.1% 

65 

30.8% 

18.5% 

Athletics/Recreational facilities 

67.3% 

74 

25.7% 

16.2% 


OIRP staff then incorporated the SSS coding into all analyses of the DSP program data. 
Existing information on seven cohorts of students (those entering UMB from Fall, 2000 to 
Fall, 2006), including admissions data, student GPAs by semester, credits accumulated by 
semester, and retention and graduation data were analyzed and compared between three 
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groups of UMB students (DSP Only, DSP & SSS, and non-DSP). A total of 1,138 DSP 
students and 3,011 non-DSP students were included in the analysis. The effect of SSS on 
retention held true in the larger analysis. 

Other factors that were analyzed showed similar results, with DSP & SSS students 
performing better than DSP Only students for the majority of cohorts. An analysis of end of 
semester cumulative GPAs showed that, overall, DSP students had lower GPAs than non- 
DSP students. However, after being coded for SSS, it was revealed that DSP & SSS students 
had average GPAs that were close to non-DSP students, while DSP Only students had the 
lowest GPAs. These results are displayed in Figures 3. and 4. below. 

Figure 3. Average GPA by Semester: DSP Students vs. Non-DSP Students 



Figure 4. Average GPA by Semester and DSP Category with SSS Coding 


Average GPA by Semester and DSP Category with SSS Coding 



SEMI SEM2 SEM3 SEM4 SEM5 SEM6 SEM7 SEM8 


More questions were then generated about what aspects of the SSS program might be 
impacting these findings. The OIRP staff was able to incorporate these questions into the 
ongoing data collection. For example, additional focus group questions were added to 
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explore areas within SSS that seem to have the biggest impact from the students’ 
perspectives. A total of 21 of the 22 students (95%) who attended a focus group and were in 
the SSS program made comments about the program, and the majority of the comments were 
positive. The SSS program seems to be effective because of its multidimensional approach 
to students. Students’ feedback centered on three main themes: 1) High quality advisors 
who both provide useful information and also personally care about the students and connect 
with them (57%); 2) Services and/or the Foundations class that are useful to students (57%); 
and 3) A space and environment that is inviting to students and creates a place to gather or 
hang out (29%). Several made comments which showed the interactive nature of the SSS 
advisors. 

“I used to have a regular advisor because I wasn’t signed up with SSS. 

You’re only a number [with university advisors], but with SSS they 
know me by my name, and you have that relationship with them. 

They know what classes you take and what you need to take. They 
know if I am missing something, and they call me. ...In other 
departments, they look at it as it’s your problem. My problems 
become our problem at SSS. They helped me fill out my FAFSA - the 
school kept telling me I made a mistake, but didn’t tell me what the 
mistake was. She [SSS advisor] actually sat down with me and filled 
it out.” 


Another student revealed about her advisor, 

“I’m really close to her. I had a problem last semester, and I talked to her. She 
understands you, she is supportive, very helpful.” 


Another student stated that even though her advisor was not always available for a meeting, 
she emails him frequently, and he responds to her emails. She also stated, “When I get 
anxious and stuff, he helps calm me down.” This feedback from focus groups led us to the 
conclusion that advising was an important factor. 

A further breakdown of the SSS services that students felt were useful revealed that of the 
17 comments that were made about the services (excluding advising), 35% were about the 
usefulness of the tutors, 29% were about the usefulness of the Foundations class, 24% were 
about the book loan program, and 12% were about the laptop loan program. One student also 
mentioned that she got a free USB port and a planner. Only three students stated that they 
did not use their advisors on a regular basis or that SSS was not helpful to them. In addition, 
students mentioned that the unlimited amount of tutoring they could get from the SSS tutors 
(versus the limits placed on University tutoring) was very helpful. 
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When survey results were analyzed, the analysis focused on study time, tutoring, and 
advising. A total of 64 students (58.2%) who returned the survey were also classified as 
Student Support Service students. These students reported the importance of and their 
satisfaction levels with subdivisions of Student Support Services programs. The highest 
percentage of students reported using their SSS advisor (92.2%), followed by the priority 
registration (89.1%) and then the cash aid/money other than financial aid loans (76.6%). The 
book loan program and the laptop loan program had the fewest percentage of students 
participating (67.2% and 68.8%, respectively), but still had the majority of students reporting 
that they used these programs. Interestingly, the two programs most used by students (SSS 
advisor and priority registration) also had the two highest percentage of students rating the 
program as very important (65.7% and 59.4%, respectively) and having been very satisfied 
with the services they received (47.5% and 43.9%, respectively), and had the lowest 
percentage of students rating that they were very unsatisfied with the service (1.7% and 
1.8%, respectively). 


Table 2. Utilization of Student Support Services Program Ranked by Importance for Success 
and Ability to Continue in School (n=64) 


Student Support Services Program 

% Used 

n Used 

% Very 
Important 

% Very 
Satisfied 

Advisor 

92.2% 

59 

71.2% 

47.5% 

Cash aid/money other than Financial 
Aid loans 

76.6% 

49 

67.3% 

30.6% 

Priority registration 

89.1% 

57 

66.7% 

43.9% 

Book loan program 

67.2% 

43 

65.1% 

34.9% 

Tutor 

73.4% 

47 

61.7% 

34.0% 

Laptop loan program 

68.8% 

44 

61.4% 

31.8% 

Foundations class 

82.8% 

53 

54.7% 

35.8% 

Referrals to other sources 

76.6% 

49 

49.0% 

32.7% 


There were no significant differences between DSP Only and DSP & SSS students on the 
amount of time spent on schoolwork each week, but higher percentages of DSP & SSS 
students reported that they completed all of their assignments (77.0% vs. 47.7%, p=.001). 
Those students who were in DSP & SSS were more likely to use the Reading, Writing, and 
Study Strategies Center (25.9% vs. 9.5%, p<.05), the subject tutoring (individual tutorials, 
including ENG 101/101E and 102/102 E) (51.7% vs. 24.4%, pc.Ol), and the ESL Center 
tutoring (40.7% vs. 7.3%, pc.OOl) than those in DSP only. 

These results led to more questions about the different types of academic advising 
available to students, and a short survey was conducted by OIRP staff with academic 
advisors from three departments (SSS, University Advising Center (UAC), and athletics) in 
order to further examine the components of the SSS advising program. Advisors were asked 
about a variety of activities that occur during advising sessions, and were asked to rate how 
frequently each activity occurs during a ‘typical’ advising session. Activities that advisors 
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say they do only sometimes or rarely include working with advisors in the students’ major 
department, performing placement tests or other assessments, meeting with or 
communicating with students’ parents, and making referrals to sources outside of the 
University. SSS advisors assist students with financial aid documents, while UAC advisors 
don’t typically do so. Among the three SSS advisors across the 26 activities (excluding 
referrals to SSS tutors), more activities were selected as being performed ‘Usually’ (n=40) 
than among UAC advisors (n=23). In addition, SSS advisors had fewer activities selected as 
‘Rarely/Never’ (n=8) than those in UAC (n=ll). 

Several advisors in the UAC stated that students get referred to them by faculty. In 
addition, the UAC advisors stated that students who are experiencing problems schedule 
appointments with them. In the UAC, a student can see any advisor, while in SSS, advisors 
are assigned a specific caseload. SSS advisors and the Athletics Department advisor have 
smaller caseloads (approximately 200 to 1) compared to the UAC advisors (approximately 
400 to 1). The SSS advisors are mandated to meet with students several times a year, as is 
the advisor in the Athletics Department. In addition, one SSS advisor and the advisor in the 
Athletics Department revealed that they teach skills courses (Foundations) or study hall to 
students, and therefore have frequent interactions with them and get to know their strengths 
and weaknesses. Several advisors also stated that they do “intrusive counseling” and 
frequently communicate through informal networks with other faculty or professional staff. 
One advisor in SSS stated that they know the students’ course schedules, and can meet up 
with students after class if they need to reach them and they are unresponsive to phone calls 
or email messages. All of the SSS advisors and the Athletics advisor said students are 
‘usually’ comfortable talking about personal, social, and academic problems, while the UAC 
advisors all stated that students were ‘often’ comfortable doing so. Perhaps the smaller 
caseloads and the relationships that the SSS advisors forge over time allow students to get 
more comfortable talking about problems with them. The SSS advisors seem to act more as 
a one-stop resource, and their lower caseloads may allow them to engage in a broader range 
of advising activities at more frequent intervals than the UAC advisors. 

All the advisors across all the departments said that students return to them even after they 
have declared a major and have a faculty or departmental advisor, because students still need 
help with general education requirements. The advisors that were interviewed saw the role 
of the faculty advisor as one of knowing specific departmental requirements and linking 
students in their majors to career opportunities. Interestingly, the SSS advisors mentioned 
the relationship that they had built with their advisees, and encouraged students to utilize 
both types of advisors. On the other hand, the UAC advisors and the advisor for student 
athletes emphasized more the distinct roles of the faculty advisor. 

One area in which DSP & SSS students did not perform as well as DSP Only students or 
non-DSP students was in graduation rates. While there were no statistically significant 
differences between the non-DSP, DSP Only, and DSP & SSS groups in the percent of 
students graduating (28.8% vs. 28.0% vs. 25.2%, respectively, p=.416) across those cohorts 
who have at least four years of data (2000-2003), there was a question as to why the higher 
retention rates of the DSP & SSS students did not translate into higher graduation rates for 
these students. The COP discussed this issue, and generated some ideas as to why this may 
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The first year college signifies a time of great transition for many students, involving 
separation from parents, immersion into a new environment, additional freedoms and greater 
responsibilities (Chickering & Reisser, 1993; Gardner, Upcraft, & Barefoot, 2005). The 
challenge posed by the college transition seems to explain why the first-year is the most likely 
time for a student to drop out of college (Gardner et al. 2005; Tinto, 1993). Research on student 
departure suggests that the first 2 to 6 weeks of the first semester represents a critical period for 
students’ decision to persist or withdraw from an institution of higher education (Levitz & Noel, 
1989). During these few weeks, the first-year students who are most likely to persist to their 
sophomore year integrate into the academic and social culture of the institution (Tinto, 1993). 

The students who successfully achieve a fit with their college or university also tend to solidify 
their financial commitment to completing an education at the institution (Paulsen & St. John, 
1997). 

Academic performance and progress represent important measures of academic 
integration (and social integration to some extent), but for first- year students the critical period in 
their transition to college occurs before the end of the first semester or even before the mid-term 
warning grades get posted. Therefore, effective retention efforts must identify the most at-risk 
first-year students prior to their first day of class in order to provide the necessary support for the 
student’s successful transition into the institution. Tinto (1993) argues that successful first-year 
programs do not focus on retention as a primary goal, but on ensuring that students acquire the 
necessary skills and habits to meet the developmental challenges that result from university life. 
The current investigation seeks to provide an effective method to identify the students who have 
the slimmest chance of returning to an institution for their sophomore year. 

Institutional researchers often utilize logistic regression to develop models that identify 
individual students with the highest probability of dropping out in an effort to provide 
individualized support for at-risk students (Dey & Astin, 1993; Hadijcostas, 2006). Retention 
prediction models for first-year persistence tend to focus on students’ preexisting characteristics 
(e.g., demographic variables, financial aid, and pre-college academic records), but do not include 


Page I 93 



measures that approximate student-faculty interaction. A vast body of literature stresses the 
importance of student-faculty interaction on student persistence beyond their first-year of college 
(Astin, 1993; Tinto, 2000). The current investigation utilizes students’ first semester course 
schedules in order to construct variables that coupled with students’ preexisting characteristics 
present a valid logistic regression model. A key finding suggests that small class size during the 
first semester (i.e., fewer than 20 students) promotes student retention from freshmen to 
sophomore year. In the discussion section, We highlight the implications of the findings for 
institutional research, academic advising, student retention, and academic affairs. 

Problem Statement 

Retention prediction models for the entering freshmen class tend to focus on students’ 
preexisting characteristics (e.g. demographic variables, financial aid, and pre-college academic 
records), but do not include measures that approximate student-faculty interaction during their 
first semester. The current study also seeks to utilize students’ first semester course schedules in 
order to construct variables that coupled with students’ preexisting characteristics suggest a 
logistic regression model that predicts student persistence beyond the first- year. The model 
utilizes variables that predict the likelihood of student persistence prior to the first day of classes 
for an entering class of first-time degree-seeking students. In addition, to predicted measures of 
faculty interaction; the current work aims replicate the findings from previous studies (e.g., that 
high school GPA and receiving financial aid predict first-year student persistence to the 
sophomore year). 


Method 

We utilized logistic regression to develop a predictive model that identified a set of 
significant predictors of persistence for the three most recent entering first-year classes. The 
model we present predicts the future likelihood (i.e. probability) that each student in the current 
first-year class would return for their sophomore year. 

Participants 

The students in the study enrolled at a moderately selective private university in the 
Northeastern United States with an enrollment around 5,000. Participants included all first-time 
full-time undergraduate degree seeking entering the university in Fall 2005, Fall 2006, Fall 2007 
and Fall 2008 enrolled at the October census date (N = 4,173). Non-U. S. citizens were excluded 
from the study. Female students made up 52% of the sample and ethnic minorities comprised 6% 
of the sample. 

Participants satisfied the federal cohort group definition (i.e., first-time first-year students 
enrolled in a degree program full-time). The prediction sample (n=3,182) consisted of the federal 
cohort groups for fall 2005, 2006, and 2007. The dependent variable (persistence to the 
sophomore year) was known for students in the prediction sample. The verification sample 
consisted of students from the fall 2008 federal cohort group (N=991), for which persistence to 
the sophomore year was not known at the time of the study and would be predicted by the model. 
The logistic regression procedure develops predictive probabilities for participants in the 
verification sample based on the significant relationships in the prediction sample. 
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Data 


Student records from the registrar, financial aid, and admissions provided the data to 
measure students’ class size, instruction from full-time faculty, financial aid (e.g., grants, student 
loans, work study), demographic characteristics (e.g., sex, race, state of residence), prior 
academic performance (e.g., high school GPA, SAT/ACT score) and whether or not students 
persisted to their sophomore year. 

Course schedules provided data on class size and instructional contact with full-time 
faculty. We examined each course schedule to calculate the percentage of a student’s courses 
delivered by full-time faculty and the percentage of the student’s course sections with 
enrollments below 20. Percentage of classes below 20 served as the measure of small class size. 
Percentage of classes taught by full-time instructional faculty served as the measure of contact 
with full-time faculty. 

Financial aid variables included whether or not a student received any form of financial 
aid (e.g., loans, grants, work study) and whether they received a Pell grant (i.e., an indicator of 
low economic status). A variable also tracked if the student had a work study position. 

Student background indicators consisted of whether a student was female or not and if the 
student was a member of a racial minority group or not. Geographic background was measured 
by whether or not the student was from one of the six New England states or not (the research 
was conducted at an institution located in New England). Prior academic performance variables 
included SAT (or ACT score converted to SAT) and high school GPA. 

The dependent variable (persistence to the second year) was measured by whether or not 
the student returned for their sophomore year. For the prediction sample enrollment data verified 
if each student returned or not for their sophomore year. For the verification sample, the model 
attempted to predict the probability of persistence for the fall 2008 federal cohort group. 

Procedures 

We used binary logistic regression as the data analysis procedure. The binary logistic 
regression procedure allows for multiple dichotomous and continuous predictor variables to 
predict a dichotomous outcome or dependent variable. In the current investigation the outcome 
variable is dichotomous (i.e., students either return for their sophomore year or they do not return 
for their sophomore year) and the predictor variables are dichotomous (e.g., sex) or continuous 
(e.g., high school cumulative grade point average). The following equation represents the logit 
equation for the persistence model in the current investigation: 

z = bo + biXj + boXi + + b^Xk 

In this equation z equals the natural log of the odds ratio. 

z = ln(odds ratio) 

The odds ratio is the probability of persistence divided by the probability of dropping out. 

Odds Ratio for Persistence = P. (Persist) 

A 

P (Dropout) 

For each significant variable the logit coefficient suggests a significant increase or 
decrease in the odds ratio. For example, if the results suggest female students are significantly 
less likely to persist to the sophomore year, the logit coefficient for female students determines 
how much the overall odds of persistence decrease if the student is female. Notice that sex is 
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only one variable in this hypothetical model, sex would actually be coupled with other 
significant variables in the model to identify students with the lowest probability of returning for 
their sophomore year. The probability that a student persists or returns for their sophomore year 
equals the following equation: 

A 

P (Persistence) = Odds Ratio 

(1 + Odds Ratio) 

To better control for error and identify the best fitting model with the fewest number of 
predictors, we utilized the hold-out cross-validation method (Oxford & Daniel, 2001). 

We split the participants into two randomly selected subsets prior to analysis. For the first subset 
of the sample, we entered all variables simultaneously in order to determine which variables 
would significantly predict persistence. The resultant model was tested on the second subset of 
the data. Significant predictor variables from the second subset established the final model, 
which was rerun against the entire study population to obtain the predicted probability of 
persistence for each student in the study. 

In order to classify the students as most likely to persist or most likely to drop-out we 
established the model’s cut-off for predicted probabilities. Traditionally, the cut-off for 
determining group membership in the dependent variable is an odds ratio of 1 to 1 or a 
probability of .50. In the case of the current study a probability of .50 indicates that students with 
a persistence probability of less than .50 are predicted to not return for their sophomore year 
while students with a probability greater .50 are likely to return for their sophomore year. For the 
purpose of identifying students who are at risk, some researchers suggests that it may be more 
appropriate to increase the sensitivity of the model to attrition by changing the cut off probability 
for persistence to .75 or an odds ratio of 3:1 (Hadijcostas, 2006). Students who have less than a 
.75 chance of returning for their sophomore year would then be predicted to depart. In the 
current work, we were more interested in identifying at risk students and decided to set the cut 
off probability for student persistence at .75 rather than .50. Thus, the institution intends to 
develop interventions for fall 2008 entering students with predictive probabilities below .75 

Results 

We initially used binary logistic regression to regress all 10 predictor variables on 
persistence in the first subset of the sample. The variables were entered simultaneously to 
determine which indicators significantly predicted persistence to the sophomore year. The odds 
ratios for each variable ranged from .62 to 1.84 (see Figure 1), but only five variables (i.e., 
female, aid received, Pell grant received, high school cumulative grade point average and 
percentage of classes under 20) significantly predicted persistence beyond the first-year p < .05. 
Being from the study institution’s geographic region (i.e., New England), SAT score (math and 
verbal total), percentage of courses taught by full-time faculty, having a work study award, and 
being a racial minority did not significantly predict persistence p > .05. 

Figure 1. First-year retention odds ratios for first-time degree- seeking undergraduate students on 
all variables. 

Note. Variables with a * significantly predict persistence at p < 0.05 
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Although the initial model with all 10 predictor variables (including 5 non-significant 
predictor variables) was not the most parsimonious model, there were no significant 
differences between observed and model-predicted values as indicated by Hosmer and 
Lemeshow goodness of fit X 2 (8, N = 1,586) = 5.49, p > .05. The initial model accounted for 6% 
of the variance in the dependent variable student persistence, Nagelkerke R 2 = .06. 

We found the five significant predictor variables from the first subset to be statistically 
significant when tested on the second subset of the sample. The five significant predictors 
supported a parsimonious model that was run against the entire sample of participants. The 
results of the final model are summarized in Table 1. 

Table 1. Variables in the final model. 



B 

S.E. 

Wald 

df 

Sig 

P- Value 

EXP(B) 
Odds Ratio 

95.0% C.I. for 
EXP(B) 

Lower Upper 

AID_Received( 1 ) 

0.632 

0.131 

23.25 

1 

0.000 

1.882 

1.455 

2.434 

Female(l) 

-0.433 

0.095 

20.94 

1 

0.000 

0.648 

0.539 

0.781 

HSGPA 

0.276 

0.114 

5.85 

1 

0.016 

1.318 

1.054 

1.649 

PELL_Received( 1 ) 

-0.446 

0.144 

9.56 

1 

0.002 

0.640 

0.483 

0.849 

ClassUnder20 

0.092 

0.042 

4.75 

1 

0.029 

1.096 

1.009 

1.191 

Constant 

0.200 

0.334 

0.36 

1 

0.549 

1.221 




Note. Variables with a * significantly predict persistence at p < 0.05 


Two significant predictors in the final model relate to financial factors. In particular, the 
odds of persistence to the sophomore year increased when students receive some kind of 
financial aid (i.e., loans, grants, or/and work study), but the odds of persistence decreased if 
students have a low enough household income to receive a Federal Pell Grant. Although not as 
strong a predictor as financial factors, being female significantly decreased the odds of 
persistence to the sophomore year. After financial factors and sex of student, high school 
cumulative grade point average was the next strongest predictor of first-year persistence. The last 
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variable in the final model to predict persistence was the percentage of small classes (i.e., 
enrollment under 20 students) a student had during their first semester of college. 

Findings suggest the our proposed final model fit the data as there were no significant 
differences between observed and model-predicted values based on the Hosmer and Lemeshow 
goodness of fit X 2 (8, N = 3,182) = 8.00, p > .05. The final model accounted for 4% of the 
variance in the dependent variable student persistence, Nagelkerke R 2 = .04 and correctly 
classified 75% of the students in the prediction sample as persisting or dropping out after the first 
year (see Table 2). 

Table 2. Classification accuracy of final model for individual cases in prediction sample. 



Predict Depart 

Predict Stay 

% Correct 

Actual Depart 

125 

473 

21% 

Actual Stay 

313 

2,271 

88% 



Overall 

75% 


Note. The cut value is .750 


Discussion 

In the current work we replicated previous findings (e.g., Paulsen & St. John, 1997) that 
financial factors have important effects on the ability of students to persist in college. Also in 
support of previous research (e.g., Kahn & Nauta, 2001), we found that high school cumulative 
grade point average positively predicts persistence beyond the first-year of college. Contrary to 
previous research we found that females were less likely than males to persist to the sophomore 
year. This finding may be institution specific and reflect the characteristics of the study 
population. The study institution has a higher first-year retention rate for males, but females have 
a higher graduation rate. 

Although the number of classes taught by full-time instructional faculty did not 
significantly predict first-year persistence, a wide body of past research suggests that student- 
faculty interaction with full-time faculty highly correlates with student persistence (Astin, 1993). 
Perhaps the percentage of classes taught by a full-time faculty does not necessary provided a 
strong enough measure of student-faculty interaction but class size does. Class size seemed to 
offer a better indicator of faculty- student interaction, given that first semester students with a 
greater percentage of small classes (i.e., fewer than 20 students) were significantly more likely to 
return for their sophomore year. The findings of the current study suggest that interaction with 
instructors fostered persistence to the sophomore year regardless of whether those instructors 
taught at the institution on a full-time or part-time basis. 

Implications and Future Research 

The current work implies that institutions with limited resources should consider utilizing 
more part-time faculty to reduce first semester class sizes. Importantly reducing class size in the 
first semester of college may better support student persistence and interaction with faculty. 
Future research should focus on the effectiveness of predictive models that incorporate variables 
from class schedules in predicting the probability that individual students will persist. This 
research suggests that early interventions can be designed for academic advisors to better assist 
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at-risk students in course selection. For enrollment managers the current investigation offers a 
method to better identify first semester students with a higher probability of leaving the 
institution. Academic Affairs should also consider student success when structuring the first-year 
curriculum. More effort to identify high risk courses for predicting student success would also be 
beneficial to institutions in developing predictive models of student persistence. 

Limitations 

A major limitation was that the study was only conducted at a single institution. Future 
studies will need to determine if the results apply to other institutions. In addition to threats to 
external validity the binary logistic regression procedure excludes participants with missing data 
limiting the variables that the current study could consider. All cases must have data for each 
variable or the case is excluded from the analysis. In the current work, all participants had data 
for each of the variables used. If participants had missing data the variable was not used. As a 
result, we did not look at a number of financial variables, such as parental income or unmet 
financial need given that a large proportion of the participants did not have data on these 
variables. However, we did consider if the student received any form of financial aid and found it 
to be a significant predictor of persistence, but the inclusion of more financial variables might 
better predict retention and further enhance the ability of the model to predict persistence. 
Institutions may need to gather information on students who do not apply for financial aid to 
better predict persistence of all students. 

Connecting Research to Practice 

The current work provided additional information into the reasons why students do or do 
not persist. The fact that a smaller class size may further increase student persistence rates 
extends an opportunity to shape a student’s class schedule during the summer orientation 
program taking into account the significant background, academic, and financial variables. We 
have coupled the findings of current work with other resources, such as the results of Noel 
Levitz’s Student College Inventory, (specifically the dropout proneness factor) in order to 
triangulate multiple sources that identify at-risk students. The information from the current work 
and other sources has been incorporated in the retention and student success efforts of the 
university. For example the university’s peer mentor program, administered through the Student 
Advocacy Office, has used this information to reach out to at-risk students in the fall 2008 
cohort, in addition to monitoring whether or not these students have received warning grades. 

The list of at-risk students has also been shared with the University’s “At-Risk” Committee, 
which works to identify students with academic and behavioral issues early in the freshmen year. 

The benefits of this research spread across the Academic and Student Affairs divisions, as 
well as the Enrollment Management & Retention division. The findings of current investigation 
have prompted us to review and refine the academic advising process in order to provide 
additional information to both the individual academic advisor and the University Advising 
Center. The at-risk student information also benefited our Student Affairs Division by helping 
them identify potential behavior and attitude issues early on. In the area of Enrollment 
Management & Retention, the early identification of at-risk students allowed us to target 
interventions for at-risk students at the outset of the student’s academic career. The Student 
Advocacy Office, which fosters monitors and collaborates on student persistence, has been able 
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to focus more closely on those students with an at-risk profile. In conclusion, the current research 
allows us to examine an additional set of variables for identifying at-risk students and work more 
efficiently to improve the overall academic success and persistence of the freshmen class. 
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be so. One of the ideas generated was that perhaps DSP & SSS students, who tend to be ESL 
students, are leaving UMB prior to having to take the Writing Proficiency Exam, which 
students are required to pass as a requirement for graduation. This area requires further 
examination. 


The Home Stretch: Creating the Action Plan 

During the course of formulating the action plan, OIRP played a critical role in presenting 
the data and leading the discussions. For example, as more and more questions arose, OIRP 
would re-direct the group to focus on the questions that could be answered with the data that 
had been obtained. When additional data was required, OIRP directed the group to focus on 
collecting data that was able to answer the questions that were generated. While many 
interesting lines of inquiry were explored, OIRP played a leadership role in clarifying the 
relationship of the data to the overall project goal. In addition, without the initiative of the 
OIRP questioning and analyzing the role of the SSS program, knowledge of the role that this 
program plays in student retention would not have become a part of the institutional 
knowledge base regarding university retention efforts. 

This iterative process produced an evidence-based intervention strategy. The plan of 
action that was created was drawn directly from the data analyses that had taken place over 
the course of the planning year. The key component of the action plan was to replicate the 
intensive advising services that are offered to SSS students, and to provide them first to DSP 
students who are not eligible for SSS. The action plan included funding a DSP Success 
Coordinator who would be responsible for implementing this strategy. Components of the 
advising were based on the research findings about SSS advising, and included maintaining a 
small caseload, mandatory advising sessions at the beginning and end of the semester, 
monitoring of student grades, teaching a college survival course, and assisting with the 
student’s financial aid requirements. In addition, at the end of the DSP Program, a new 
survey was incorporated in which students conducted a self-assessment of their skills and 
indicated the areas in which they would want additional help. The DSP Success Coordinator 
would review the student’s self-assessment of services required. In addition, the DSP 
Success Coordinator would work with representatives from Career Services and individual 
Colleges and Departments to host a series of career development workshops, and would work 
more closely with departmental advisors, two components that the data revealed could be 
more pronounced in the current SSS advising model. The plan also included a research 
fellow position that would be responsible for assessing the strengths and weaknesses of the 
new model, so that subsequent adjustments could be made. In addition, the research fellow 
would examine in more depth the question of why higher retention rates for DSP & SSS 
students were not translating into higher graduation rates. Overall, the intervention was 
designed to capitalize on the parts of the SSS advising model that the data pointed to as being 
helpful from the students’ perspectives, and to introduce some new elements that were 
needed in some important areas. 

Decisions were also made to exclude some components from the action plan. One 
example was the priority registration element. Priority registration was used by most of the 
students (89.1%), and ranked third in importance, perhaps because it allowed students to get 
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into their top choice of classes. However, the long term goal of the intervention was to 
eventually incorporate the successful components university-wide, and priority registration 
was incompatible with that goal, as it cannot be applied universally or it defeats the purpose. 
In addition, financial assistance was ranked second in importance, but was not implemented 
as part of the intervention. The reason for this decision was because the focus group data 
revealed that students had a lot of confusion regarding the financial aid component of their 
education, and the COP felt that this area needed to be explored further. Provisions were 
made instead for the DSP Success Coordinator to review student financial aid packages and 
for the Research Fellow to examine the financial aid piece in more detail. 

Implications for IR Professionals 

In conclusion, there are many ways that the findings of this research project can be helpful 
for other IR professionals. First, the project documents a process whereby IR leadership in 
the research planning and implementation stages created value-added data and analyses to 
program leaders. Data were collected and utilized in a way that would allow it to directly 
support a plan of action with a specific goal, and the iterative process utilized to move data 
into action is outlined for others. Historical IR data had to be combined with current data 
collection and information from program leaders in order to make sense. Otherwise, the role 
of the SSS program on retention would have remained submerged within the existing 
datasets. In addition, qualitative data from the focus groups added a layer and texture to the 
analysis that helped tease out some important issues and add perspective to other issues. 
Focus group results revealed that the tutoring and advising component of the SSS program 
were significant aspects of assistance to students, and interviews with advisors shed more 
light on the differences between general university advisors and advisors in the SSS program. 
Last, this collaborative IR model, including a funded research fellow position, is now being 
extended to other departments on campus. Plans are in place for a second year of funding for 
a research fellow to examine assessment issues in the Student Affairs division. This 
structure will be explored further as a novel way to include short-term yet extensive data 
projects from various departments under the IR umbrella. 
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